Samuel Williams Monday, 11 June 2012

My application Property Manager uses CoreData for its document model and storage. Because the Mac App Store is now requesting that applications are sandboxed, I recently worked on updating Property Manager to work correctly in the sandbox.

Problem Identification

As Property Manager acquires new features (in this case, additional tagging and notation facilities), the data model needs to be updated. This in turn depends on CoreData to migrate the user's previous documents to the latest data model. Traditionally, CoreData migrates the data transparently when the user opens the file, however here lies the problem with sandboxing: the updated file cannot be saved to disk unless the user is presented a NSSavePanel. Because of this, automatic migrations will fail.

As a secondary problem, CoreData only migrates from one version to the next, either by inferring the mapping model or by selecting a single mapping model. If a user migrates from v2 file format to v3, CoreData will handle this automatically. However, CoreData will not handle the case where the user has v1 data which needs to be converted to v2 and then finally v3. In general this seems like very fragile behavior for an application that is managing important user data.

Solution

It turns out that the solution to the above problem is fairly involved, and I ended up rewriting some code (originally written by Marcus S. Zarra in his book Core Data).

In order to kick off the new migration process, we need to hijack the document creation process and check if migration is required:


@implementation PMDocumentController

- (id)makeDocumentWithContentsOfURL:(NSURL *)url ofType:(NSString *)typeName error:(NSError **)outError {
	// Initialize the migration controller which will be used to figure out if migration is required:
	PMMigrationController * migrationController = [[PMMigrationController new] autorelease];
	
	// This is the target model we are using:
	NSURL * targetModelURL = [[NSBundle mainBundle] URLForResource:@"PropertyManager" withExtension:@"momd"];
	migrationController.targetModel = [[[NSManagedObjectModel alloc] initWithContentsOfURL:targetModelURL] autorelease];
	
	// Other details such as the store type and all the intermediate models we could migrate to:
	migrationController.storeType = NSBinaryStoreType;
	migrationController.intermediateModels = [PMMigrationController findIntermediateModelsForBundle:[NSBundle mainBundle]];
	
	// If migration is required:
	if ([migrationController requiresMigration:url]) {
		// Show the user a save panel and tell them that they need to save the migrated document
		NSSavePanel * savePanel = [NSSavePanel savePanel];
		savePanel.title = @"Document Migration";
		savePanel.directoryURL = url;
		savePanel.nameFieldStringValue = [url lastPathComponent];
		
		savePanel.allowedFileTypes = [NSArray arrayWithObject:typeName];
		savePanel.allowsOtherFileTypes = NO;
		
		savePanel.message = @"Your document needs to be upgraded.";
		
		[savePanel setCanSelectHiddenExtension:YES];
		[savePanel setExtensionHidden:YES];
		
		if ([savePanel runModal] == NSFileHandlingPanelOKButton) {
			// At this point, the sandbox has allowed us access to the file given by savePanel.URL
			
			// Perform the migration:
			if ([migrationController migrateURL:url toURL:savePanel.URL error:outError]) {
				// Hide the file if requested:
				[[NSFileManager defaultManager] setAttributes:[NSDictionary dictionaryWithObject:[NSNumber numberWithBool:savePanel.isExtensionHidden] forKey:NSFileExtensionHidden] ofItemAtPath:savePanel.URL.path error:nil];
				
				// Migrate and open new document:
				return [super makeDocumentWithContentsOfURL:savePanel.URL ofType:typeName error:outError];
			}
		}
		
		// outError should have been filled in by migrationController!
		return nil;
	} else {
		return [super makeDocumentWithContentsOfURL:url ofType:typeName error:outError];
	}
}

- (id)makeDocumentForURL:(NSURL *)urlOrNil withContentsOfURL:(NSURL *)contentsURL ofType:(NSString *)typeName error:(NSError **)outError {
	// ...
}

@end

Finally, the migration controller code incrementally migrates from one model to the next until it gets to the targetModel.


// Interface:

@interface PMMigrationController : NSObject

@property(nonatomic,retain) NSString * storeType;
@property(nonatomic,retain) NSManagedObjectModel * targetModel;
@property(nonatomic,retain) NSArray * bundles;
@property(nonatomic,retain) NSArray * intermediateModels;
@property(nonatomic,retain) NSURL * temporaryDirectory;

+ (NSArray*)findIntermediateModelsForBundle: (NSBundle*)bundle;
+ (NSURL*) temporaryDirectoryAppropriateForURL:(NSURL*)sourceURL;

- (BOOL) requiresMigration:(NSURL*)storeURL;
- (BOOL) migrateURL:(NSURL*)sourceStoreURL toURL:(NSURL*)destinationStoreURL error:(NSError **)error;

@end

// Implementation:

@implementation PMMigrationController

@synthesize storeType = _storeType, targetModel = _targetModel, bundles = _bundles, intermediateModels = _intermediateModels, temporaryDirectory = _temporaryDirectory;

+ (NSArray*)findIntermediateModelsForBundle: (NSBundle*)bundle {
	//Find all of the mom and momd files in the Resources directory
    NSMutableArray * modelPaths = [NSMutableArray array];
	
	NSAutoreleasePool * pool = [NSAutoreleasePool new];
	
	// Scan model directories:
	NSArray * modelDirectories = [bundle pathsForResourcesOfType:@"momd" inDirectory:nil];
    for (NSString * path in modelDirectories) {
        NSString * subdirectory = [path lastPathComponent];
		
        NSArray * array = [bundle pathsForResourcesOfType:@"mom" inDirectory:subdirectory];
        [modelPaths addObjectsFromArray:array];
    }
	
    NSArray* otherModels = [bundle pathsForResourcesOfType:@"mom" inDirectory:nil];
    [modelPaths addObjectsFromArray:otherModels];

	[pool release];
	
	return modelPaths;
}

- (NSMappingModel*) mappingModelForSourceModel:(NSManagedObjectModel*)sourceModel targetModel:(NSManagedObjectModel**)targetModel {
	NSMappingModel * mappingModel = nil;
	
    for (NSString * modelPath in self.intermediateModels) {
        *targetModel = [[[NSManagedObjectModel alloc] initWithContentsOfURL:[NSURL fileURLWithPath:modelPath]] autorelease];
        mappingModel = [NSMappingModel mappingModelFromBundles:self.bundles forSourceModel:sourceModel destinationModel:*targetModel];
		
        if (mappingModel) {
			NSLog(@"Found migration model: %@", modelPath);
			// If we found a mapping model then proceed:
            return mappingModel;
        }
    }
		
    // We have tested every model and didn't find any candidates:
	*targetModel = nil;
	return nil;
}

- (NSMigrationManager*) fetchMigrationManagerForSourceMetadata:(NSDictionary*)sourceMetadata mappingModel:(NSMappingModel **)mappingModel error:(NSError**)error {
    //Find the source model:
    NSManagedObjectModel *sourceModel = [NSManagedObjectModel mergedModelFromBundles:nil forStoreMetadata:sourceMetadata];
    if(sourceModel == nil) {
        NSLog(@"Failed to find source model %@", [sourceMetadata description]);
        return NO;
    }
	
	NSManagedObjectModel * targetModel = nil;
	*mappingModel = [self mappingModelForSourceModel:sourceModel targetModel:&targetModel];
	
	if (*mappingModel == nil) {
		NSLog(@"No mapping model found!");
		return NO;
	}
	
    return [[[NSMigrationManager alloc] initWithSourceModel:sourceModel destinationModel:targetModel] autorelease];
}

+ (NSURL*) temporaryDirectoryAppropriateForURL:(NSURL*)sourceURL {
	NSError * error = nil;
	
	NSURL * intermediateURL = [[NSFileManager defaultManager] URLForDirectory:NSItemReplacementDirectory inDomain:NSUserDomainMask appropriateForURL:sourceURL create:YES error:&error];

	if (error) {
		NSLog(@"Error finding temporary directory for URL %@: %@", sourceURL, error);
	}
	
	return intermediateURL;
}

- (BOOL) progressivelyMigrateURL:(NSURL*)sourceStoreURL toURL:(NSURL*)destinationStoreURL step:(NSUInteger)step error:(NSError **)error {
	// Get the source metadata:
    NSDictionary *sourceMetadata = [NSPersistentStoreCoordinator metadataForPersistentStoreOfType:self.storeType URL:sourceStoreURL error:error];
	
    if (!sourceMetadata) {
        return NO;
    }
	
    if ([self.targetModel isConfiguration:nil compatibleWithStoreMetadata:sourceMetadata]) {
		// We have migrated to the target model, move the sourceStoreURL to destinationStoreURL:
		NSFileManager * fileManager = [NSFileManager defaultManager];
		
		if ([fileManager fileExistsAtPath:[destinationStoreURL path]]) {
			[fileManager removeItemAtURL:destinationStoreURL error:error];
		//	[[NSWorkspace sharedWorkspace] recycleURLs:[NSArray arrayWithObject:destinationStoreURL] completionHandler:nil];
		}
		
		return [fileManager copyItemAtURL:sourceStoreURL toURL:destinationStoreURL error:error];
    }
	
	// Figure out how to do the migration:
	NSMappingModel * mappingModel = nil;
	NSMigrationManager * manager = [self fetchMigrationManagerForSourceMetadata:sourceMetadata mappingModel:&mappingModel error:error];
	
	if (!manager) {
		NSLog(@"Failed to instantiate migration manager!");
		return NO;
	}
	
	// Actually do the migration:
	NSURL * intermediateStoreURL = [self.temporaryDirectory URLByAppendingPathComponent:[NSString stringWithFormat:@"migration-%d", step]];
	
	NSLog(@"Migrating URL %@ to %@", sourceStoreURL, intermediateStoreURL);
    if (![manager migrateStoreFromURL:sourceStoreURL type:self.storeType options:nil withMappingModel:mappingModel toDestinationURL:intermediateStoreURL destinationType:self.storeType destinationOptions:nil error:error]) {
		NSLog(@"Migration failed!");
        return NO;
    }
	
    // Rince and repeat:
	return [self progressivelyMigrateURL:intermediateStoreURL toURL:destinationStoreURL step:step+1 error:error];
}

- (BOOL) requiresMigration:(NSURL*)storeURL {
	NSError * error = nil;
	
	// Get the source metadata:
    NSDictionary *sourceMetadata = [NSPersistentStoreCoordinator metadataForPersistentStoreOfType:self.storeType URL:storeURL error:&error];
	
    if (!sourceMetadata) {
		NSLog(@"Error fetching source metadata: %@", error);
        return NO;
    }
	
    if ([self.targetModel isConfiguration:nil compatibleWithStoreMetadata:sourceMetadata]) {
		return NO;
    }
	
	return YES;
}

- (BOOL) migrateURL:(NSURL*)sourceStoreURL toURL:(NSURL*)destinationStoreURL error:(NSError **)error {
	self.temporaryDirectory = [[self class] temporaryDirectoryAppropriateForURL:sourceStoreURL];
	
	// We copy the source to avoid any problems with source and destination being the same:
	NSFileManager * fileManager = [NSFileManager defaultManager];
	
	NSURL * initialURL = [self.temporaryDirectory URLByAppendingPathComponent:@"initial"];
	if (![fileManager copyItemAtURL:sourceStoreURL toURL:initialURL error:error]) {
		NSLog(@"Failed to create initial migration source, copying %@ to %@: %@", sourceStoreURL, initialURL, error);
		return NO;
	}
	
	BOOL result = [self progressivelyMigrateURL:initialURL toURL:destinationStoreURL step:1 error:error];
	
	// Ignore errors...
	[fileManager removeItemAtURL:self.temporaryDirectory error:nil];
	
	return result;
}

@end

Sandboxing Problems

I would like to say that the above code works flawlessly. In fact it does, under a non-sandbox environment. At this point in time, I've found various bugs and problems with NSSavePanel when used in the sandbox, although these problems don't seem to be officially acknowledged by Apple. The biggest road block I ran into is that essentially savePanel.URL is always nil in the sandbox environment... Hopefully this will be fixed in a future release.

However, at the very least, migrating more than one model works perfectly, and the user is given the opportunity to save their datafile in a new location which at least means that the data won't be damaged by a failed migration.

Comments

Hi Samuel,

Is there a trick to where the mapping model should be? or a naming convention that should be used for the mapping models?

I keep triggering the “No mapping model found!” NSLog.

I know the models work since I can migrate when using temporary entitlements. I just can’t figure out how to make the program manually find them.

Hi Jo3rw,

You can debug the migration process by adding the launch argument -com.apple.CoreData.MigrationDebug 1 to your application. It should print out quite a bit of information.

In my case, my PropertyManager*-*.xcmappingmodel files are listed in the “Compile Sources” build phase for my application, so I can only assume they end up in the main application bundle.

The error message you are receiving is given when there is no migration found from the current Core Data model to some other Core Data model. This may happen if you accidentally modified one of the intermediate managed object models. I suggest you start with the above debug information and try to find out how far it is getting through the migration process.

Feel free to provide more details and I’ll see what I can do to help.

Kind regards,
Samuel

Hi Samuel,

Thanks for helping. In the app I have now, I am running two mapping models to go from version 1→2→3. Using your debugging suggestion I found half of my hash tags in both the source and destination do not match.

ie: Version 1 test File Source does not match 1st Mapper Source (1→2)
Version 2 test File Source does not match 2nd Mapper Source (2→3)

I’m guessing this is why I get “No Mapping Model found”. Is there a way to fix the hash tags?

@Jo3rw The hash tags come from the models themselves. You might want to try recreating any mapping models with invalid hash tags.

Thanks for the sample code. In my sandboxed app I too had the problem of savepanel.url returning nil. After some experimenting, I thought I’d try passing the same URL to both arguments of the migrationController in makeDocumentWithContentsOfURL like this:

if ([migrationController migrateURL:url toURL:url error:outError])

And it worked! No savepanel needed!

I should also mention that is being discussed on the Apple dev forums.

Thanks for the good work. It helped me a lot. Two things I find out:

First, the save panel delivers url as nil in a sandbox. It looks like, that the sandbox does not like to initialize the directory url with a file path. You have to set the following:

if ([url isFileURL])
{
	homeURL = [url URLByDeletingLastPathComponent];
}

savePanel.directoryURL = homeURL;
savePanel.nameFieldStringValue = [url lastPathComponent];

and it will work now.

At at second, you have to set the storeType explicit to NSSQLiteStoreType, if you use SQLlite storage. This worked fine for me.

Leave a comment

Please note, comments must be formatted using Markdown. Links can be enclosed in angle brackets, e.g. <www.codeotaku.com>.