SPM Import Process
SPM Import Process This document discusses the overall procedure and specific layouts needed to import data from a foriegn system into SPM.
Go into the requests area of building maintenance and make sure all the request priorities that you expect to use/need are setup correctly and in order. This is VERY important!
Note: I wouldn't do this unless the your about to schedule. If converting an existing system, it's not worth the hassle
Note: Must import course catalog & master schedule first
Note: If using anything but the default schedule status code, set
them up *BEFORE* importing or you'll be sorry!
Note: Must import student schedules first
Note: Must import student schedules first
Now is a good time to prove the building enrollment in the building maintainence against what the district think it has. Also, proof your history by running a enrollment history report for the year. The final enrollment figures should match the buildings enrollment.
Since transcripts span multiple years, there is a bit more setup involved in importing them into the system.
The most important thing to remember is that you will need to create a school year for each years worth of transcripts you plan to import. However, as the act of creating a school year copies off the nearest existing school year to it, you MUST make sure that school year is complete and correctly setup (grading periods, grading setup, course/level grades, etc.
IF you create the additional school years and then realize you need to change something (list of grade, grading periods, marks, etc), you will have to remake the change to each school year, one at a time - yuck.
Save yourself a lot of time and don't even start the transcript conversion and school year creation until the main/first school year has everything correct in it!
All import programs work by reading a simple ASCII text file and importing it's contents into SPM.
Each line of the file is terminated with an appropriate line delimiter for the operating system (a simple line feed for unix, a carraige return and line feed combination for MSDOS and VMS systems).
Each field in the file is delimited from the next field by a single space. All data *can* be quoted with double quote characters, but only fields that contain spaces must be quoted. An empty field can be specified by either a 'quote/space/quote' sequence (that's three characters like " ") or a single, unquoted dash (-) character.
If you need to include a double quote as part of a field, you can 'quote the quote'. For example, a field that looked like
"The outboard ""motor"""
would be imported as
The outboard "motor"
When an importer has a problem importing a line, it will print the
line # of the problem line, possibly with some additional information,
into the import log file. You
Once the importer starts working, it will display a status box that shows the number of lines/records to import from the source file, the number of lines already imported, the %age done, the number of errors, the number of lines per second, the elapsed time and the estimated time remaining.
The estimates are just that - estimates. The actual time will very depending on the complexity of each line, the speed of the system, etc. Typically, the estimated time will stabilize after the first 100 or so records are processed.
Most importers ask similar questions for importing. A typical (though not complete) set of prompts might be:
School Year to import to: 1994/95
Import from: student.imp
Purge students first: No
Existing records should be: Kept
Log import results to: student.log
Auto-create codes: No
Backup replaced data to: student.org
This is the school year the importer will use. You can import multiple school years into SPM, but for simplicity sake, it's best if you restrict the process to one school year at a time.
This is the file containing the data to be imported. When you fill it in, most importers will then provide a default for the 'Log import results to' and 'Backup replaced data to' files. This file must conform to the rules mentioned earlier.
The word 'students' may be replaced with something more appropriate to the type of data being imported. The question allows you to first purge all instances of the records you are about to import before the actual importation process begins. You would not normally need to do this unless you were re-doing a previous bad import and wanted to start with a fresh slate. Note that this can take quite a while!
This allows you to direct what the importer will do when faced with attempting to import a record already in the SPM database.
If set to Kept, a duplicate in the import file will result in a warning message in the log file and the duplicate being ignored. In that case, nothing at all occurs to the previously existing record in the database.
If set to Replaced, the existing record, if any, will be replaced by the duplicate record. If the importer has a 'Backup replaced data' feature, the original record will be 'exported' to this backup file before being replaced. This provides a way to re-import the backup file in case you want to return to a pre-import status.
For import records that do not already exist in the database, this option doesn't do anything.
This is the name of a log file that will be used to record any errors or warnings from the import process. A default name is filled in as soon as you fill in the 'Import From' file, but you can override it.
You MUST always check this file after completing an import. You should understand each and every error message and whether the message indicates something you need to correct BEFORE moving to the next import step.
Fair Warning - I have no pity on anyone who does an import and does not scrupulously check the log files. In fact, if you have a problem, the first things I'm going to want to see is the log files from the import.
Lines in the log files usually tell you if the associated import record was not imported. If there was an error while importing a record, the record is typically not imported (the import transaction is backed out) and the log file has a message ending with the words "NOT imported". Warnings typically say 'WARNING' at the begning or just don't have a 'NOT imported' message after them.
When the importers are allowed to create ancillary records as part of the import process, they will log any created record to the log file with a note that you should check the code out. DO NOT IGNORE THIS RECOMMENDATION. After the import, you should (actually MUST) use the appropriate maintenance to look over any newly created codes, put meaningful descriptions in and tailor the contents of the created codes/records.
Sometimes an error may cause earlier data to be undone. This usually happens when the importer has started processing an import record, create a few new codes the record needs and soon after, the importer discovers a fatal error that prevents the import record from being saved. In that case, the log file may have several messages about records being created, then a message about an error that was not imported. If you then look for a code that was supposedly created in SPM, you may not find it because when the import error occured, everything associated with that imported line was 'undone'.
When set to yes, the importer will try to create any codes it needs to complete the import process. For example, when importing a student record, the importer will look up the grade the student is supposed to be in. If that grade is not found and the 'Auto create' option was set to Yes, the grade record will be automatically created. When a code is autoatically created, a message is printed in the import log file telling you about the creation and asking you to look into it. You really want to do this as the importer creates codes, but usually has pretty ugly ones.
If you answer No to this question and the importer comes across an import record with a code not already on file, the importer will reject the record, not import it and print an error message in the log file indicating that the import record used an unknown code.
There are some things the importer will never create. These tend to be big-ticket items like school years, buildings, etc - things that general are too complicated to just be created on the fly.
Not every importer has this option. Basically, this works in conjunction with the 'Existing records should be:' option. If the user sepcified that existing records should be replaced, then the importer will 'export' a copy of any existing database record to this file before replaing it with the duplicate import record.
The 'exported' record is written using the same import format the importer program expects. This means that if you later want to recapture the replaced data, you can just re-run the importer and tell it the 'Import From:' file is the backup file created by an earlier import.
Becaue of the importance of this data (in terms of ability to recover from a mistake), if you specify a file that already exists, the importer will ask you if you really want to use the file if there is already an existing file by that name.
This field has a default name that is filled in automatically when you select the 'Import From:' file. You can always override the name.
The student master importer is imp-student.p and is responsible for basic demographic information about a student as well as things like the building, grade, home room, etc of a student.
The student importer will ask you for a 'Historical Date of Import'. This is used for creating student activity/history records to record the act of being imported. The date should be a reasonably accurate one.
The importer will create (as needed) ethnic code records, grade (K-12) records, room catalog records, status code records, special ed code records, personnel records, promotion path records and building house records if the 'Auto Create:' option is enabled.
The student note importer is imp-note.p and is responsbile for importing the student note data.
The note importer will create note codes (if you enable the option), but will not create note groups. It will also create site tables and table coes, if they are supplied and 'Auto Create:' mode is enabled.
The student contact importer is imp-contact.p and is responsbile for importing the student contact data.
The contact importer will create (as needed) contact codes and mail class records - if the 'Auto Create:' option is set to Yes.
The 'Mailings' field is a bit different than most. It controls which mailsing a contact should receive. The two easy values are "All" and "None" and they do pretty much what you would expect.
You can also specify a comma delimited list of mailing class names. There shouldn't be any spaces in the list. If the list specifies mailing class names that are not already defined and the 'Auto Create:' option is set to Yes, then the mailing class record will be created. Specifying a list of mailing classes is equivilent to setting a contact to recieve 'Some' mailings in the student maintenance.
The student medical importer is imp-medical.p and is responsible for importing medical notes.
The medical data importer will (if needed) create medical codes, if the 'Auto Create:' option is set to yes.
The student Daily attendance importer is imp-daily.p and is responsbile for importing student daily attendance data.
The daily attendance importer will (if needed) create daily attendance codes and student contact records (if the 'Auto Create:' option is set to Yes)
The attendance date must be a valid date marked as a membership date in the attendance calendar.
The student course requests importer is imp-rqst.p and is responsible for importing student requests.
Unless you have a special case, you should leave the course credit, request FTE and priority fields empty (-). You want them to use the course catalog definitions whenver possible.
The student schedules importer is imp-schd.p and is responsbile for importing the student scheduled courses.
Unless you have a special case, you should leave the status code, grading policy and attempted credits fields empty (-). You really want them to use the master schedule definition.
Possible values for Grading Policy are "Pass/Fail", "Numeric" or "Letter". As stated earlier, unless you REALLY have to, leave the grading policy field blank and let the master schedule control it.
The student course attendance importer is imp-crsatd.p and is responsbile for importing course and period based attendance.
You must have previously imported the students schedule before importing the course attendance data.
The course attendance importer will (if needed) create course attendance codes and student contact records (if the 'Auto Create:' option is set to Yes)
The course and section #'s must be both in the master schedule and in the student's schedule. The attendance date must be a valid date marked as a membership date in the attendance calendar.
Unless you need to import the period # explicitly, leave it blank. If left blank, the period # will come from the master schedule. Even if you supply a period #, it must be a period the specifed course meets in. You really want to leave this field blank.
The student report card mark importer is imp-grades.p and is responsible for importing student report card marks and comments.
You must have previosuly imported the student schedule before importing the report card data. Also, you must have setup the marks via the grade setup maintenance, setup the grading periods and the actual letter/numeric grades a student can receive via the course/level grades maintenance.
Grades are defined in the same order that they are enabled in the grading period maintenance for the period(s) being imported. All grades are letter gradres, regardless of how the master schedule or students schedule is setup (they are internally converted into whatever format is needed).
Comments are optional. If supplied, they are numbers and should already be defined (the importer will create them, if needed and 'Allow Create:' is set to Yes). Since there are 5 comments and 5 grades, it may appear that they are ordinally related, they are not. There are upto 5 grades for a course/grading period and upto 5 comments for the same course/grading period.
The course and section #'s must be both in the master schedule and in the student's schedule.
The student transcripts importer is imp-tran.p and is reponsible for importing student transcript records.
Transcripts are considerably more complicated that most other items being importer. You should be sure to have read the Overview of the transcript import process before continuing.
You must have a school year created for each school year you intend on importing. Each of the school years must have (at a minimum), all buildings and grades, a set of grading periods, a course catalog (ideally a master schedule and even better is a full set of student schedules), a correctly setup grading setup, and a full set of course/level grades.
To make your life easiest, make sure these are all setup in the 'oldest' school year you have on file before creating even older school years. If you have it all setup, the data will be copied over for you when the older school years are created. Then you either have little or nothing to do with those old years.
The transcript importer imports an entire school year record for a given student and course at a time. As a result, the order that the marks for the course are listed in is dependent on the enabled marks for all grading periods of the school year.
For example, a 2 semester, 4 marking period district would interpret the imported marks like this:
Import fld #: 13 14 15 16 17 18 19 20
Mark Name: MP1 MP2 EX1 SM1 MP3 MP4 EX2 SM2
Grades/Marks can be left blank, but you must leave the field blank and not shift everything over. For example, if a district only has semester grades to import, the import text would look like (for the above setup):
Import fld #: 13 14 15 16 17 18 19 20
Mark Name: MP1 MP2 EX1 SM1 MP3 MP4 EX2 SM2
Marks: - - - A+ - - - C
You MUST have at least one mark for the course or it will not be imported.
The importer has a 'prune' option. When enabled, after import is complete, it will run through all the transcripts and eliminate any student transcript course records that have no marks. After that, it will eliminate any student transcript years with no courses and finally will eliminate any student transcripts without any years.
This SHOULD be run at some point in the import to clean up potentially illegal transcripts. If you are importing in several batches, you should probably defer this until the final batch to prevent accidental deletion (although, anything deleted and referenced in a later import would be recrated with no info loss, so even this is too much a considerations).
The format of the import record is:
Unless you have a really good reason, you probably want to leave the 'Rankable' and 'Attempted Credits' fields blank and let the system use the master schedule (or course catalog) defaults.
The student history note importer is imp-history.p and is responsible for importing notes for historical purpose.
History notes differ from regular notes inthat they contain change and state of the student data from a specific point in time. They don't actually affect the contents of any other student records, they just provide a place to keep convient student historical notes.
If needed, the history note importer will create history code records, as long as the 'Auto Create:' option is set to yes.
The student behavior importer is imp-behave.p and is responsible for importing student behavior incidents and associated student behavior events.
If needed, the behave importer will create behave code and behave action records as long as the 'AutoCreate' option is set to yes.
The personnel catalog importer is imp-prsn.p and is reponsible for importing personnel data and assigning the imported records to appropriate buildings.
Unlike many other importers, you can have multiple import records for the same person - providing the building, department or house (if the building uses houses) is different (any one or any combination). This allows you to have people who exist only once in the system, but have multiple assigments. Since a person has to be assigned to a building before they can be used in that building, this lets you consolidate your personnel catalog to make personnel maintenance much easier.
The personnel catalog importer will, as needed, create the department, house, home room and personnel class - provided the 'Auto Create:' option is enabled.
Personnel codes must be unique.
The room catalog importer is imp-room.p and is responsbile for importing room definitions for each building.
The home room personnel code, if specified, is the personnel code of the person who is considered to be responsible for the room. The department will be, as needed, created if the 'Auto Create:' option is set to Yes.
The course catalog importer is imp-crscat.p and is responsible for importing the course catalog.
The course catalog importer will create, as needed, the course group, department and course grading level if the 'Auto Create:' option is set to Yes.
You REALLY want to leave the grading policy, course priority and course grading level blank unless the course explicitly needs them.
The 'Blocked Course' allows you to define either a stand-alone course, a master course that will be the "header" for a new blocked course or define the course a being a child of a master course.
Leave it blank for a completely stand-alone course. Set it as 'MASTER' if the course is the definition of a blocked course (i.e. this is the 'header' course). If this course is part of a blocked course, then include the course # of the master here. The referenced course must already exist and be set up as a MASTER course.
The master schedule importer is imp-master.p and is reponsbile for importing the master schedule.
You must have already imported the course catalog before beginging the master schedule import.
The master schedule importer will create, as needed, the scheduling group, personnel code, course grading level, scheduling period and room if the 'Auto Create:' option is set to Yes.
Note that there can be multiple occurances of a given master master schedule with different meeting #'s (#20) to allow mote than 1 meeting for a course to be loaded.
You REALLY want to leave the grading policy and course grading level blank unless the course explicitly needs them.
The Address/carrier route importer is imp-address.p and is reponsbile for importing the optional address/carrier route file to allow for address validation.
Each imported record can cover a single address (high and low street numbers are the same) or a range of addresses for a single street. There can be multiple ranges for each street, though there should not be any overlap. The Street side indicator is most useful when planning on doing bulk postal mailings. If you are using this strictly for address validation, you can use the "B" (Both sides) code.
Many fields are optional and really only useful when doing bulk postal mailings or voter lists. The external codes 1-5 are there for district use - SPM will not use them and does nothing with them, other than storing them and allowing you access to them.