Career and Technical Education Audit – Frequently Asked Data and Methodology Questions
This document addresses Frequently Asked Questions regarding the Florida Department of Education’s (FDOE) annual Career and Technical Education (CTE) audit data and methodology. Questions arose in the weeks and months following the dissemination of the local level CTE audit data and data webinar in fall 2021. The CTE audit data webinar and state and local level data Excel workbooks are located on the Office of CTE Quality website linked here: Career Pathways.
If you have additional questions concerning the CTE audit data webinar or the CTE audit state and local level data, please reach out to us at Keith.Richard@fldoe.org and Brittany.Salaam@fldoe.org. Please ensure that Brittany.Salaam@fldoe.org is included in all emails regarding methodology questions to ensure we have logged your question.
General CTE Audit Data Questions
1.	Why are three years of data reported in the CTE audit files?
Due to the reporting lag of student records, the most recent data in the CTE audit excel files pertain to the 2019-20 Reporting Year (RY). These data were the most recent available at the time of running these reports. Historical data are also included in the excel files (2017-18 and 2018-19 RYs) to allow districts and colleges to analyze longitudinal performance trends. In the coming months, the FDOE will be working to provide an annual update on these data to include the 2020-21 RY.
2.	The CTE audit state and local level data was disseminated to the field on 09/24/2021. Have there been any updates to the files since dissemination? 
Yes, there have been updates to the Postsecondary CTE audit state and local level data files. The new files are found in the “Correspondence” section on the CTE audit website Career Pathways. Direct links to these data files are below:
2022-01-19 Postsecondary CTE Audit Summary
2022-01-19 Postsecondary CTE Audit Detailed  
3.	What data have been updated in the “new” CTE audit state level and local level data files (mentioned above)? 
There have been updates to the retention data for the district technical colleges in the postsecondary CTE audit data. This means the statewide aggregated numbers for these data have also changed (i.e., retention metrics aggregated statewide to the 10-digit CIP or Program Code). In our continued effort to provide accurate and clean data, our office identified an issue that impacted some of the “Retention” columns in the state level and local levels files of the POSTSECONDARY CTE audit workbooks.
Issues were found in the Retention numerator columns for the years “2016-17 Retained into 2017-18,” and “2017-18 Retained into 2018-19,” on the state level and local level workbooks. District technical colleges are the only sector in which the error was found; the error was not associated with local level data for FCS institutions. There have been no changes to the data for any Secondary files (state level or local level) and no changes to any FCS postsecondary (local level) data.  
The Postsecondary summary workbook and the Postsecondary detailed workbook (and the corresponding state level and local level sheets) have been updated to reflect accurate data in the newly revised files posted on the CTE audit website (see links in question #2 above).
4.	Is there a recording of the “CTE audit state level and local level Data Pull” webinar?
A copy of the CTE audit state level and local level Data Pull Webinar recording can be located on our website: Career Pathways under the “Correspondence” section, titled “2021-10-25 Webinar on CTE Audit Local and State Level Program Data.”
5.	Will Year 2 (this year) of the CTE audit follow the same model as Year 1 (last year), including phase 3 that required feedback from districts and colleges? 
Year 2 (this year) of the CTE audit will NOT replicate Year 1 (last year) of the CTE audit. Year 2 of the CTE audit is strictly an informational year and will NOT include a phase 3 that required local level feedback on the CTE audit data. Therefore, no additional action is required from any institutions or districts. Rather, we recommend you review the data locally to identify high performing programs and programs requiring improvement.
6.	The four CTE audit Excel Workbooks serve as Year 2 of the CTE audit (2020-21). Where can copies of the CTE audit state and local level data used in Year 2 be located?
A copy of the CTE audit state and local level data Workbooks can be located on our website: Career Pathways, in the “Correspondence” section (see question #1). There are four Excel Workbooks: two Postsecondary workbooks and two Secondary workbooks. Each Excel Workbook contains three worksheets: a data dictionary sheet, a local level data sheet and a state level data sheet.
See next question for additional details. 
7.	What are the differences between the four Excel Workbooks? 
There are two Postsecondary Workbooks and two Secondary Workbooks. Each educational level (postsecondary and secondary) has a “detailed” workbook of the data and a “summary” workbook of the data. The detailed version of the workbooks include all the data our office used for the calculations in the CTE audit, including the denominator and numerator headcounts used to generate the various performance percentages. The summary version provides a condensed version of the data and essentially only reports the final performance percentages for each CTE audit metric.
See next question for additional details. 
8.	What is the difference between the summary report and the detailed report data? Does it provide the same exact data? 
The summary report and detailed report vary in the amount of information shown pertaining to the CTE audit process. However, for the columns that do appear in both versions, the data are exactly the same. 
9.	What should our office do if we notice programs in the CTE audit local level data workbooks that are no longer being offered in our district/institution? 
We recognize that programs listed for a given district or college may not currently be offered by the district or college. Rows of programs were reported for a given district/ college if at least one student was found to be associated with that program in the reporting window used to pull the CTE audit data (which included several years of reported data). Therefore, some program rows in the file may be included for a program that was previously offered locally but is no longer offered, or is perhaps in “teach out” locally. The FDOE does not currently maintain a postsecondary CTE academic program inventory and therefore we were not able to filter out programs that are not currently being offered in a given district or college. However, the FDOE is currently working on establishing an active academic program inventory for postsecondary CTE programs per section 35 of House Bill 1507. In the future, we will use this academic program inventory to filter out programs that are not active in each local context.
Secondary CTE programs that are not offered by a given district may appear on the CTE audit file for additional, different reasons. To calculate the annual “Concentrators Percentage” in the secondary CTE audit, students are tracked for all of their secondary CTE course-taking behavior in the current year and previous six years of course taking behavior (this flags student as a “Participant,” which is a headcount that serves as the denominator in the Concentrator Percentage calculation). When doing so, this essentially counts students in a program for which they may not actually be enrolled, but rather, they were simply enrolled in a course related to the secondary CTE program. When this occurs, a program row is added to the CTE audit file for a given district. Importantly, some secondary CTE courses are aligned with more than one secondary CTE program. Therefore, some programs may never have been fully offered (or existed) in the district, but it would have appeared on the CTE audit file if there was a single course enrollment in a course that was embedded in the secondary CTE program.
Please see question #35 for related information. 
10.	How is program 10-digit CIP code (or Program code) used in the CTE audit state and local level data workbooks? Is the 10-digit CIP code (or Program code) updated when changes are required?  
The 10-digit CIP code (or Program code) information is generated by the Division of Career and Adult Education, maintained in the “CIP table” of approved statewide program frameworks by the Program and Policy Coordinator. This includes the CIP number, the Program Title, the Program Type, and Career Cluster. 
This means that the 10-digit CIP code (or Program code) and the associated Program Title will reflect what is listed on the state’s approved curriculum frameworks. We recognize that locally, some districts or colleges may use a slightly different Program Title in their local program inventories than what is listed on the approved curriculum framework, and that is okay. However, these programs will share a 10-digit CIP code (or Program code) and are therefore the same program of study.
11.	Are programs evaluated on the CTE audit metrics by tracking individual students?
All metrics for the CTE audit (e.g., Retention, Success, etc.) are calculated by tracking individual level student administrative records, to see if each individual student meets the specific CTE audit metrics. These data are then aggregated up to the program level, either for a local context (district or college) or for the entire state level (all programs across all districts and colleges combined and aggregated to the program level). The CTE audit tracks individuals based on the business rules and technical specifications for each CTE audit metric.
12.	Where are the student level data for the CTE audit state level and local level data collected from? 
Educational student level data used in the CTE audit are submitted by school districts and Florida College System (FCS) institutions as part of the regular annual processes for other statutory reporting requirements. These data are then utilized for various accountability reports, including the annual CTE audit.
The PK-20 Education Reporting and Accessibility (PERA) team is an office in the FDOE. PERA uses statistical coding software to compile student level data analyzed in the CTE audit. The PERA team has access to these data from the FDOE data collection offices and databases, including Community College & Technical Center Management Information Systems (CCTCMIS) for Florida College System student data and Workforce Development Information System (WDIS) for district CTE student data.
13.	There are several different reports conducted by the WDIS team. Which of these reports is used for the CTE audit? 
District data are pulled in from the (WDIS) annual “Survey 5” report. Survey 5 refers to the district level data collection of final, end-of-year information on secondary career and technical education and industry certifications. The survey is conducted and finalized in August of each year. 
For additional information please visit WDIS section of the FDOE webpage. 
14.	CCTCMIS and WDIS collect data throughout the year and changes are sometimes made to submitted data. Does the CTE audit use the finalized “closed” data or does the CTE audit use data submitted during the “open” reporting time frames?  
FCS data are pulled from Term 1E, Term 2E and Term 3E data reports after the (CCTCMIS) has closed the final annual data submission process. In other words, the final, closed end-of-year submission data are used to pull the student record-level data in the majority of the CTE audit calculations for FCS student data. 
The final, closed end-of-year Survey 5 submission data are used to pull the student record-level data in the majority of the CTE audit calculations for district student data. Postsecondary district technical college data and district secondary CTE data are collected by the Florida Department of Education in the format of surveys of school district student and staff information in scheduled times during the reporting year. The reporting year begins with Survey 1 in August and concludes with “closed/ finalized” data in the following August at the end of the reporting year. Below is a list of surveys conducted in order by month the data is due for the reporting year.
Survey 1: August
Survey A: August
Survey 6: September
Survey 8: July through September
Survey M: September
Survey B: October
Survey 2: October
Survey 9: December
Survey C: January
Survey 3: February
Survey N: February
Survey D: March
Survey 4: June
Survey 5: August
15.	Are there any instances when the data used in the CTE audit calculations come from surveys or terms that are still “open” reporting time frames? 
Yes, there is one instance when the closed reporting information is not used for the CTE audit. This is when tracking students for the “continuing education” metrics in the CTE audit, and the decision to use “open, not finalized” data in this calculation matches the business rules for all other “continuing education” reports (as in, reports other than the CTE audit) generated by the FDOE’s Florida Education and Training Placement Information Program (FETPIP) office to track “continuing education.” 
When pulling data to calculate “continuing education,” the FETPIP office uses the most recently available data, which includes the “open” beginning of spring term data. This is because these are the most currently available data at the time these reports are generated on an annual basis. More specifically, when tracking a student for “continuing education” in the CTE audit and in all other standard FDOE reports, a program completer is tracked into the next reporting year, where “next reporting year” is operationalized as the subsequent fall end-of-term and spring beginning-of-term data submissions. 
For example, a 2019-20 program completer is tracked for continuing education in the 2020-21 reporting year fall end-of-term and spring beginning-of-term data submissions. In this example, the 2020-21 reporting year data is still open and can be modified by districts and FCS institutions at the 2020-21 close data submission date (in this example, July or August 2021).
16.	Students who complete a program of study are tracked for employment post program completion. Where is this information collected from? 
Employment information for each program completer is maintained and reported by the (FETPIP) team at FDOE. Employment status (yes or no) and quarterly occupational earnings of program completers are available to the FETPIP group using Florida’s Unemployment Insurance Wage Record data. These data are made available to the FDOE by a three-way data sharing agreement between the Department of Revenue, Department of Economic Opportunity (DEO) and the FDOE.
17.	The CTE audit state level and local level data workbooks show occupational projections data, including occupational openings, occupational growth, and entry-level occupational wages. Where are the occupational wage data collected from?  
These data are different than the data used by the FETPIP office. These occupational projections data are based on occupations (SOC codes) and not based on tracking Unemployment Insurance Wage Records of educational program completers (see question #16 for additional details). The occupational projections data are calculated by DEO and are publicly available on their website. These data are incorporated into the CTE audit, to allow us to compare the earnings of our program completers (FETPIP data) to the earnings we would expect them to earn in a given occupation (DEO data). 
To make these comparisons, the CTE audit assigns a “Primary SOC code” (Standard Occupational Classification code) to each educational program of study. The data calculated by DEO for each SOC code (each occupation) include the following data fields: entry level wage, mean wage, annual number of job openings, annual percent job growth. These four data fields (calculated by DEO) are included in the CTE audit as part of the institutional program performance metrics and the labor market demand metrics. Further, DEO generates these calculations (for each SOC code) at the state-level and regionally for each of Florida’s 24 Local Workforce Development Area (LWDAs).
18.	Are the wage data provided by DEO calculating the wages of actual program completers found in a given district/college program?
No. A large section of the CTE audit files reference wage and occupational data based on the Standard Occupational Classification (SOC) code, which are generated by DEO. These data are based on employment projections for using survey data that includes Floridians in a given SOC, regardless of whether they graduated from an associated CTE program of study or not. As in, the DEO projections data are not simply based on district or college program graduates in that occupation. These DEO projections wage data (for a given SOC code) are paired to a given program of study based on the Primary SOC associated with each educational program of study, using the extant CIP-to-SOC crosswalk maintained by the FDOE. 
See question #17 for additional details.
Alternatively, data exclusive to district and FCS program completers are generated by the FETPIP office within the FDOE. Data generated by FETPIP track the actual quarterly earnings annualized for district and FCS programs completers using Unemployment Insurance Wage Records. 
See question #16 for additional details.  
19.	If the occupational wage information is pulled from DEO and are not specific to a district/ FCS program, how is the middle wage metric for the CTE audit determined? 
To determine if a program of study met the “middle-wage threshold” (one of the four Labor Market Demand indicators in the CTE audit), only DEO data were used (as in, FETPIP data are not used for this CTE audit calculation). 
First, the thresholds for middle- and high-wages were established using DEO data found the Statewide Demand Occupation List and the 24 Regional Demand Occupation Lists. For example, for middle-wage thresholds, one threshold was established for the state level, and 24 different thresholds were established for each of the 24 local workforce regions. Further, the “middle wage threshold” includes a threshold for entry-level SOC wages and a threshold for average level SOC wages.
These DEO established thresholds were then compared to the DEO calculations for occupational entry level and occupational average level earnings (based on SOC codes), where a single primary SOC code was associated with each of the educational programs of study. If the DEO occupational wage projections for entry level and average level wages for a given SOC code exceeded the “middle wage threshold” criteria, then the educational program aligned to that SOC code was flagged as “meeting the middle wage demand threshold” (Labor Market Demand indicator #4).
20.	First, what criteria were set for a program to be flagged as “high growth” in Market Demand Indicator #3? Second, were the criteria for "high growth” the same for local level and state level analyses? 
The criteria for a program to be considered "high growth” differed based on local level and state level analyses. 
Local level criteria for "high growth” were: (1) A positive annual average local Growth Rate, as well as (2a) At least 30 local annual Job Openings (for small Local Workforce Development Areas) or (2b) At least 80 local annual Job Openings (mid/large Local Workforce Development Areas).
State level criteria for “high growth” had two different possibilities for meeting the criteria:
•	The first option (referred to as Option A) criteria required: (1) A positive annual average Growth Rate; and (2) At least 1,200 Annual Job Openings.
•	The second option (referred to as Option B) criteria required: (1) An annual average Growth Rate of at least 1.26%; and (2) At least 500 Annual Job Openings.
21.	One of the CTE audit performance measures tracks "employment” among program completers through the FDOE’s FETPIP office. Do these data account for students only if they are employed in the State of Florida? 
Yes, the CTE audit data only includes employment rates and/or quarterly annualized earnings for program completers employed by a Florida employer. Currently, Florida’s out of state employment data sharing agreement, known as the “SWIS” (State Wage Interchange System) agreement does not allow the CTE audit to use any out of state information.
FETPIP is able to track employment and/or quarterly earnings of program completers through a data sharing agreement with the Florida Department of Economic Opportunity by accessing Florida’s quarterly Employer Unemployment Insurance (U/I) Wage Records. The Florida U/I Wage Records are exclusive to Florida employers. Therefore, federal employees who reside in Florida will not be captured in these data. Although federal employment data were once available to FETPIP through the United States Office of Personnel Management, this federal data sharing system went offline in 2018.      
22.	Does the CTE audit student employment data tracked by FETPIP account for program completers employed in the U.S. military? 
No, students who are employed by the U.S. military are not included in the performance metric that tracks employment and/or wage data for program completers. Although military data were once historically available to FETPIP from the Department of Defense (DoD), new DoD data have not been available for FETPIP’s tracking purposes since September 2015.
In sum, the CTE audit does not track federal, military or out of state employment. See question #21 for additional details.
23.	One of the CTE audit performance metrics tracks “continuing education” for program completers. Does the CTE audit data track students continuing their education outside of the State of Florida?  
No, students continuing their education outside the State of Florida are not captured in this performance metric by the FETPIP office. Currently, there are no intentions to further accommodate the tracking of students continuing their education outside of Florida.
24.	Currently, all the data provided in the CTE audit workbooks is state and local (regionalized) based information. In the future will the data provided be specific to a school or institution?  
Presently, there are no plans to disaggregate the data at the specific schools within a given district. CTE audit data and data metrics will continue to be disaggregated at the Postsecondary (district/FCS) and Secondary levels.
25.	How does Market Demand for the CTE audit correlate with Perkins CLNA? 
Programs that are flagged for not initially meeting labor market demand in the CTE audit can then be analyzed using the Perkins Comprehensive Local Needs Assessment (CLNA) Labor Market Alignment (LMA) submitted to the FDOE by each district and institutions that receives Perkins funding. 
Even if a program did not meet the Labor Market Demand indicators in the first set of CTE audit analyses (Statewide DOL, Regional DOL, High growth, Middle-to-high wages) found in the publicly disseminated CTE audit excel files, programs are considered to have met “market demand requirements” for the CTE audit if market demand was demonstrated on the CLNA LMA for a given district or college (not shown on the publicly disseminated CTE audit files). These programs were reviewed by FDOE staff using the Perkins CLNA LMA documentation already submitted to the FDOE for Perkins reporting compliance. When reviewing the CLNA information for the audit, we sought to identify the supporting evidence that demonstrated local labor market alignment for a given program. While the CLNA does have upcoming changes taking place, this does not currently affect the use of it for the CTE audit. 
For additional information on the CLNA changes, please visit the Florida Department of Education CLNA page. 
26.	The CTE audit uses SOC codes to align programs with DEO’s demand occupational list (DOL) for Market Demand alignment. Please explain how this process is conducted.
This is based on a historical list of CIP-to-SOC code alignments established by the Bureau of Standards, Benchmarks & Frameworks within the Division of Career and Adult Education at the FDOE. New program frameworks undertake a three year review process and once approved they are assigned a SOC code for the program. The FDOE is working on a process to improve and revise the CIP-to-SOC code alignments and additional information will be provided as this process develops. 
27.	The CTE audit uses several different resources to develop the data metrics. How is the data received and organized for each metric in the CTE audit to ensure clean data is provided to districts and FCS institutions? 
The Bureau of PK-20 Education Reporting and Accessibility (PERA) receives all data from the aforementioned resources (CCTCMIS, WDIS, DEO, and Division of Career and Adult Education) and creates cohorts to analyze metrics for the CTE audit data report.
The office of CTE Quality does not alter any of these data reports for the CTE audit. The CTE audit analyses the student record level data submitted to the FDOE, using various business rules and criteria established for each of the CTE audit metrics. Therefore, please be aware that concepts that sound similar (e.g., Retention) may be measured differently when comparing CTE audit data to data found in other FDOE reports.
28.	FERPA compliance is important when distributing data. Are the state and local level data workbooks FERPA compliant?
Yes, all state and local level CTE audit data disseminated meet FERPA compliance requirements. For our reporting purposes of externally shared data, FERPA compliance requires that if the denominator contains fewer than 10 students then the data (both denominator and numerator) must be masked.
The exception to this requirement occurs if the denominator has 10 or more students, the associated numerator does not need to be masked (even if it is fewer than 10 students). If the denominator has fewer than 10 students, both the denominator and the numerator will be masked to maintain FERPA compliance.
29.	How can the freeze panes feature be used when reviewing the data provided in the CTE audit Excel workbooks? 
The Secondary and Postsecondary workbooks do have the capability to “freeze” excel panes for ease in viewing your district or institution's information. Here’s how:
The files were disseminated with the “Page Layout” display selected so that the FDOE logo and document header would appear in the workbook. Please see the icons listed in the bottom right corner of the Excel window to determine which display layout is active. In order to freeze panes in a given excel sheet, change the “Page Layout” option to the “Normal” layout option (found in the bottom right corner of the Excel workbook). This will remove the sheet’s header and make the Freeze Panes feature accessible for use.
Please note the reference to the bottom right corner of the Excel window is applicable to Microsoft® Excel® for Microsoft 365 MSO (Version 2110 Build 16.0.14527.20234) 64-bit.
30.	In House Bill 1507, it states that the “Commissioner of Education is to conduct an annual review of CTE audit programs.” How does the CTE audit work conducted by the “Office of CTE Quality” assist in achieving this requirement? 
The CTE audit (Year 1 and Year 2) is the review required to be conducted in accordance with House Bill (HB) 1507 enacted in 2021. 
The review was strictly informational in Year 2 (this year) of the CTE audit. The report generated will not be used to recommend phasing out programs that provide unwarranted program duplications or do not meet institutional performance or labor market demand criteria.
For additional information on the CTE audit review mandated by Florida Law, please refer to s. 1003.491(5), F.S.
31.	Section 22 1003.491(5)(c), F.S., provides for the programs in institutions/districts failing to meet established criteria. Can you provide the current status of this work?  
The FDOE (Office of CTE Quality) initiated the rulemaking process to determine a future use for CTE audit reports being reviewed by the Commissioner. These reports will be used to make recommendations for phasing out programs that provide unwarranted program duplications or do not meet institutional performance or labor market demand criteria. The CTE audit process, data pull, and metrics will be updated to meet the changes and requirements set forth in the rulemaking process. This is forthcoming, with additional details to be provided. 
On Friday, January 21, 2022, the Division of Career and Adult Education held the CTE Program Performance Reporting rule (6A-10.0342) development workshop. A recording of the CTE audit rule workshop and the presentation slide deck is located on the CTE audit webpage in the Correspondence Section: Career Pathways.
Since initiating the rulemaking process, the department has postponed drafting a proposed version of the rule because the Framework of Quality and the Master Credentials List is still in development by the Credentials Review Committee. 
Please note, the department has also recently disseminated a Frequently Asked Questions document pertaining to the CTE audit rulemaking process. Please refer to this document for additional information. 
32.	What types of changes should districts and institutions be prepared to see with the implementation of the CTE audit rulemaking process?
The CTE audit rulemaking will identify programs that require strategic improvement plans to enhance the quality of programs provided to CTE students in the State of Florida. In addition, rulemaking will identify and report suggested programs for removal or replacement that do not align with Florida’s Market Demand or are unwarranted duplication.

Secondary CTE Audit Data Questions
33.	The CTE audit data file tracks Florida Career and Technical Education programs throughout the state. Which CTE Secondary program types are reviewed in the CTE audit data file?
The Secondary Program Types reviewed by the CTE audit data file: 
•	Career Preparatory programs; and
•	Technology Education programs
34.	Can you explain how the participation metric in the CTE audit measures student participants?
For the “participation” metric of the CTE audit, students are counted if the student receives at least 1 credit on their transcript record for a course associated with a secondary CTE program. 
The student must have been enrolled in the secondary program either in the current metric year or any of the six previous years. The student must receive at least 1 high school credit on their transcript record for a course associated with that program (.5 (half) credits or non-completers are not honored for the participant metric).
Please see question #9 and #35 for related information. 
35.	If the enrollment and participant data does not match for the program, can you explain how we would have program participation data but no program enrollment data for a given program? 
Yes, a program may have participant data and no enrollment data if any of these occurrences were found: 
•	The student was enrolled in the secondary program at another district and received at least .5 credits on their transcript for a course in the program. Then the student transfers districts but continues in the CTE secondary program and earns another .5 course credit on their transcript in the new district. The student will be “counted” in the participant headcount in BOTH districts that the student participated in. 
Or
•	The program has been phased out and is no longer offered by the district but was previously offered in any of the last six years beginning with the reporting year column. If the student met all the requirements as a participant the student is counted in the participant column even if the program is no longer being offered.  
For example, if a student is enrolled in a secondary program and the student earns a full course credit in the welding program in 2014-15. Then the welding program is “phased out” in the subsequent year of 2015-16. The student will be counted in the participant metric for the secondary program in the 2017-18 column. However, since the program has been phased out and is no longer being offered there will be no enrollments for the program in the reporting year of 2017-18 or any of the corresponding reporting school years (i.e., 2018-19 or 2019-20). 
•	If the student was enrolled in the secondary program at another district and received at least .5 credits on their transcript. 
Please see question #9 for related information. 
In order to provide better clarity on what is being calculated, we will rename these columns in future versions of the CTE audit data file. 
36.	How are secondary CTE students categorized into different programs for the CTE audit?
A student’s program of study (reported by the district to the FDOE) is used as the primary identifier to categorize students into different CTE programs for the following CTE audit metrics:
•	Program enrollments; and
•	Industry certifications (the number of students enrolled in a program who earned at least one industry certification in the Reporting Year).
A program of study record is determined by the Vocational Program Code reported by the district for that student on the Career and Technical Education Student Course Schedule. Valid Vocational Program Codes found in the CTE audit must also be included in the DCAE inventory that includes all Career Preparatory and Technical Education program frameworks that are currently active (including program frameworks that are in teach out).
Alternatively, a student’s course taking behavior (reported by the district to the FDOE), in combination with their program of study (reported by the district to the FDOE), are the primary identifiers to categorize students into different programs for the following CTE audit metrics:
•	Concentrators;
•	Concentrators who graduated; and
•	Concentrators found continuing education or employed.
Finally, a student’s course taking behavior, without referencing their reported program of study, is used as the primary identifier to categorize students into different programs for following CTE audit metric:
•	Participants: The participant data field in the CTE audit data files should not be evaluated independently. Rather, “participants” is a headcount of students that is used as the denominator to calculate the percentage of students that are “Concentrators.”
37.	Programs with rolling enrollment start dates and various completion dates can run across school years, how is this accounted from in the CTE audit? 
Survey 5 is used to capture a full Reporting Year (summer, fall, and spring) of district CTE student data. We did not make any adjustments for rolling enrollment start dates or completion dates that run across school years. If a student had an enrollment or completion record in surveys F, W, or S (DE 175825) within a particular WDIS Reporting Year (DE 186999) at the time of the final data submission (Survey 5), they were included in the metric for that year.
38.	Districts are familiar with seeing their secondary district program level data on the “O-file.” The CTE audit data does not reflect the information provided in the O-file. Why is the CTE audit data different than what was submitted in the O-file? 
The O-file is a requestable report for districts and colleges—from a separate office in the FDOE—allowing the district to review student data submitted by the district to the FDOE in a given year, matching to a specific survey. The O-file is an excellent resource for cleaned district data. The CTE audit does not include the O-file as a resource because the O-file does not use any of the CTE audit’s internal business rules for calculating metrics specific to the CTE audit. In other words, the O-file could be considered a cleaned data file without the same parameters that are used capture students in the various metrics of the CTE audit. CTE audit data follows a specific set of business rules and technical specifications. These rules and technical specifications track students based on metrics and calculations created for the CTE audit by the CTE audit steering committee in 2019 and follow guidelines set forth by s. 1003.491(5), F.S.  
Therefore, metrics (i.e., industry certification, completions, etc.) may appear to be similar between the O-file and the CTE audit, but these are not necessarily based on the same inclusion or exclusion criteria.
39.	There are instances when our district does not offer or require students to take the first course on the FDOE framework. In these cases, are the students being counted in the “concentrator” metric for the CTE audit?  
Yes. Students are not required to follow the FDOE framework of course sequencing to be counted in the concentrator metric for the CTE audit. The concentrator metric is based on a count of courses taken, not on the order in which courses are taken. In other words, the sequences of courses does not determine if a student is a concentrator or not for the CTE audit. The audit reviews seven years of data (the current Reporting Year and six previous years). If the student meets the credit criteria (see criteria below) they are counted as a concentrator:   
•	3 or More High School Credits;
•	1 or More High School Credits and 3 or More Dual Enrollment Credits;
•	2 or More High School Credits (in Programs Requiring Only Two Credits); or
•	1 or More High School Credits (in Programs Requiring Only Two Credits) and 2 or More Dual Enrollment Credits.
40.	How are students in various enrollment and completion time frames accounted for in each metric (e.g., Retention, Success, etc.) of the CTE audit?
The CTE audit uses “closed” data from Districts survey 5 responses to account for a full year-census of each program. Therefore, if a student shows up in the school year they are included in the appropriate metric.
41.	Our district offers programs that share entry level courses. Once a student has completed an entry level course he/she may branch into a separate program. How are programs with multiply pathways tracked in the CTE audit?
This question is best suited under the context of measuring “concentrators” in the CTE audit. The CTE audit tracks student level data based on course enrollments, and links these course enrollments to different programs of study. Therefore, a student can be flagged as a “participant” in multiple CTE audit programs that share the same courses. However, a student is only flagged as a “concentrator” in each program if the student has the required number of courses (see question #36 for additional details), and the student has a program enrollment record along with those courses. 
42.	If a student earns an industry certification in 2018-19 but was enrolled in the previous school year (2017-18), will the student be counted in the 2018-19 industry count or the 2017-18? Can you explain how industry certifications in the CTE audit are measured? 
In order for a student to be included in the industry certification count the student is required to have been enrolled in a secondary program in the same year that the student tests AND passes, earning the industry certification.
In other words, the student in the question scenario will not be counted in the 2018-19 or the 2017-18 school year. This is due to the student not being enrolled, not testing and not earning the industry certification in the same year.
See question #43 for further explanation on what the industry certification counts reflect.  
43.	Can you explain how industry certifications in the CTE audit are measured? The industry certification counts look “off” and are not the same numbers that we submitted for CAPE funding. Why is this the case?
The Industry Certification columns in the CTE audit do not reflect the total number of industry certifications earned in a given Reporting Year. Rather, these columns reflect the total number of students enrolled in the program each year who earned one or more industry certifications in that year related to the program. In other words, these numbers represent student headcounts and do not represent counts of industry certifications. 

Postsecondary CTE Audit Data Questions
44.	The CTE audit tracks Florida Career and Technical Education programs throughout the state. Which CTE Postsecondary program types are reviewed in the CTE audit data file? 
The Postsecondary Program Types reviewed in the CTE audit data file are:
•	Apprenticeships;
•	Career Certificates;
•	Applied Technology Diplomas (clock and credit-hour);
•	College Credit Certificates;
•	Associate in Science Degrees; and
•	Associate in Applied Science Degrees.
45.	How are completion rates measured for Postsecondary CTE program completers? 
The CTE audit does not technically have a “completion rate” metric. Please refer to the data definitions worksheet in the “Postsec CTE audit_2020-21_detailed” excel file or “Postsec CTE audit_2020-21_summary” excel file for information on the annual program completers count for the audit. 
Rather than a “completion rate” or a “150% normal-time-to-completion rate,” the CTE audit data reports the annual number of program enrollments and program completions for each program. These numbers should not be divided by each other to generate a “completion rate.” This is because the enrollments and completions included in the CTE audit data file are based on each annual Reporting Year, and these headcounts do not account for when a student begins their program.
Alternatively, the CTE audit does include what we refer to as a Success rate. In the CTE audit, Success rate is defined as the percentage of students who leave their postsecondary education sector with either a degree or certificate completion, an industry certification earned, or an occupational completion point completed. 
Please see question #46 for related information.
46.	How is “success” measured in the CTE audit? If a student completes a program or certification, are they included in the Success metric? 
The Success rate metric tracks students who “left the FCS system or left the district technical college system.” In the CTE audit, this is defined as students who were enrolled in one year of the CTE audit and not found enrolled in the next year (e.g., found in 2018-19 but not found in 2019-20). Among the cohort of students who “left,” the “Success” metric counts students who “left” with any one of the following: degree or certificate completed, or an industry certification earned, or an occupational completion point completed.
47.	When calculating my institution’s Success rate percentage, I am unable to replicate the percentage data found in the workbooks. Please explain how the Success rate percentage is obtained for the CTE audit? 
Please note, it is not appropriate to use the annual “Enrollments” and annual “Completions” headcount columns to calculate the Success rate. These columns are included as pieces of contextual information, but these are not performance metrics. See question #46 for additional details. 
The Enrollments and Completions columns cannot be used to calculate the success rate because Enrollments and Completions count the annual number of students enrolled in a program and the annual number of students that completed the program. The Success rate, on the other hand, is an entirely different metric. Please see the data definitions tab in the CTE audit data excel workbooks for a full description of this metric. 
Please see question #45 and question #46 for additional details.
48.	A student who completes a College Credit Certificate (CCC) program can employed and enrolled in the related Associate in Science program the following year. Which CTE audit metric does the student get counted for, continuing education or employed? 
In the CTE audit, the “continuing education” and/or “employed” status is combined into a single metric. In this scenario, the student earning the CCC will be identified by FETPIP as BOTH employed and as continuing education. The “Continuing Education or Employed (Best Wage)” metric in the CTE audit covers BOTH continuing education and employment for students. Therefore, in this example, the student will be counted once in this metric and receive credit for their achievement. 
49.	Which survey was used to account for students’ enrollment based on the last program of study for the Retention metric? 
The Retention metric in the CTE audit measures year-to-year retention, and includes all currently enrolled students (in other words, this metric is not a fall-to-fall retention, it is not exclusive to first-time-in-college students and it is not exclusive to the first year a student was enrolled in a program of study). 
For example, a student will be flagged for their enrollment in a program in the 2018-19 Reporting Year (RY) and then tracked into the 2019-20 RY to determine if they were enrolled in any postsecondary program (not necessarily the same program of study). In this example, 2018-19 refers to the initial year (although not necessarily the student’s first year in the program) and 2019-20 refers to the follow-up year. Further, in this example, students who were not “retained” in 2019-20 because they completed a program of study in 2018-19 are still counted favorably as a “retained” student in this metric. 
In the example above, we recognize that students may be enrolled in more than one program in the Florida College System (FCS) in the initial year for tracking year-to-year retention (e.g., 2018-19 RY). Therefore, in the FCS data, when flagging a student’s program of study, we opted to only count the last program of study a student was enrolled in from the initial year for tracking (e.g., 2018-19 RY).
For example, if the student was enrolled in a College Credit Certificate (CCC) program in fall of the 2018-19 RY, but was subsequently enrolled in an Associate in Science (AS) program in spring of the 2018-19 RY, the CCC program is ignored and only the AS program is flagged in 2018-19 and then tracked into 2019-20. 
Note, these data are based on the student’s last program of enrollment in a given reporting year, only after the End-of-Year Reporting window has closed.
Alternatively, the data were pulled slightly different for the district technical colleges. In the district technical college data, all of a student’s programs of study within a given RY were tracked for the Retention metric. In other words, if a student in the technical college system was enrolled in a Career Certificate program in fall 2018-19 RY and then a different Career Certificate in spring 2018-19 RY, both of those programs were tracked into the 2019-20 RY for the Retention metric. Please note, because Retention in the follow-up year counted any program of study (not necessarily the same program of study), programs were not penalized if a student switched their programs (and had more than one program enrollment) during the initial year.