Perhaps you're using low level configuration settings which are file-related. In such cases it's recommended to use the database configuration for the defragmentation process. Especially settings like string-encoding and block-size need to be configured properly for the defragmentation.
DefragmentConfig config = new DefragmentConfig("database.db4o"); // It's best to use the very same configuration you use for the regular database final EmbeddedConfiguration configuration = Db4oEmbedded.newConfiguration(); config.db4oConfig(configuration); Defragment.defrag(config);