Dean Herbert
9ed0e8891b
Merge branch 'fix-nested-folder-migration-endless-copy' into migration-ui
2020-05-15 13:04:16 +09:00
Dean Herbert
0690d81bbb
Add protection against migrating to a nested folder
2020-05-15 10:40:26 +09:00
Dean Herbert
25bbb02999
Throw better exceptions from OsuStorage
2020-05-15 10:28:03 +09:00
Dean Herbert
364aa5aa12
Add protection against migrating to a nested folder
2020-05-14 22:45:03 +09:00
Dean Herbert
827d75b152
Revert "Add protection against migrating to a nested folder"
...
This reverts commit ef8375b442
2020-05-14 22:44:27 +09:00
Dean Herbert
ef8375b442
Add protection against migrating to a nested folder
2020-05-14 22:42:42 +09:00
Dean Herbert
06f507496a
Delete migration source if no files exist after completion
2020-05-14 17:41:55 +09:00
Dean Herbert
cb0b25ac55
Throw better exceptions from OsuStorage
2020-05-14 17:41:55 +09:00
Dean Herbert
ad1d050fb4
Throw exception on copy timeout
2020-05-13 20:29:15 +09:00
Dean Herbert
49e616b7e5
Also check for directory presence before migrating
2020-05-13 20:19:14 +09:00
Dean Herbert
9de216f1b4
Merge branch 'master' into migration-backend
2020-05-13 20:15:22 +09:00
Huo Yaoyuan
f07d95ac59
Use IReadOnlyList<T> for TypedListConverter.
2020-05-12 21:49:55 +08:00
Huo Yaoyuan
f7d26688f4
Merge branch 'master' into json-converter
2020-05-12 21:43:02 +08:00
Dean Herbert
e4a23b3e7d
Fix ignored excluding more than top level
2020-05-12 12:39:04 +09:00
Dean Herbert
984f27c107
Add simple retry logic on file copy failure (may be in use)
2020-05-11 21:39:44 +09:00
Dean Herbert
a11be07bb1
Move log storage location after migration complete
2020-05-11 21:39:44 +09:00
Dean Herbert
36640f5dda
Merge branch 'master' into migration-backend
2020-05-11 18:52:36 +09:00
Dean Herbert
384862d48b
Fix incorrect relative paths when using GetStorageForDirectory
2020-05-10 13:20:13 +09:00
Dean Herbert
bbebd26efb
Use DirectoryInfo in more places
2020-05-09 20:13:31 +09:00
Huo Yaoyuan
80a193a616
Use IEnumerable<T> for TypedListConverter.
2020-05-09 17:41:59 +08:00
Huo Yaoyuan
74cbe9306c
Use strongly-typed JsonConerter.
2020-05-09 16:39:46 +08:00
Dean Herbert
7a2020fd45
User copy operation instead of move
2020-05-07 22:12:24 +09:00
Dean Herbert
49a03f1c06
Add basic blocking migration, move not copy
2020-05-07 22:11:57 +09:00
Dean Herbert
7781408643
Update in line with framework storage changes
2020-05-06 18:28:24 +09:00
smoogipoo
d27ca725f9
Use IEnumerable<T> instead
2020-04-22 09:04:44 +09:00
smoogipoo
a702a521f8
Fix not being able to serialise converted beatmaps
2020-04-22 09:04:44 +09:00
smoogipoo
f6f5de7ad1
Allow LineBufferedReader to keep underlying stream open
2020-03-31 10:13:50 +09:00
Dean Herbert
022465f546
Add encoding and import support
2020-03-24 14:51:52 +09:00
Dean Herbert
68ebe98fde
Remove unused GetUnderlyingStream method
2020-03-24 14:08:25 +09:00
Huo Yaoyuan
b14c7bd171
Use type switch in SearializationWriter.
2020-01-27 21:49:34 +08:00
Huo Yaoyuan
40b43b85f1
CA1715: use prefix for generic parameters.
2019-12-17 13:00:25 +08:00
Huo Yaoyuan
e46f6627e4
CA1052: make type static.
2019-12-17 13:00:13 +08:00
Huo Yaoyuan
31cc0d13da
Use 'out var'.
2019-11-12 19:55:26 +08:00
Dean Herbert
35be8f9dfb
Share framework file-exclusion function
2019-10-30 19:34:17 +09:00
Bartłomiej Dach
4b84564f47
Switch casing comparison mode to ordinal
...
Switch from InvariantCultureIgnoreCase to OrdinalIgnoreCase when
checking file paths in archives for substrings indicating the file can
be ignored for performance gains.
Co-Authored-By: Dan Balasescu <smoogipoo@smgi.me>
2019-10-11 09:24:41 +02:00
Bartłomiej Dach
57bfa18359
Filter out OS-generated files from archives
...
Add a filename ignore list to ZipArchiveReader to filter out superfluous
OS-generated files from archives during the import process. In addition
to decreasing the size of files imported this allows imports of some
incorrectly-constructed archives. An example is the case of having
a __MACOSX directory next to a single directory with the actual files -
filtering out the former at ZipArchiveReader allows the fallback added
in #6170 to work.
2019-10-10 23:54:23 +02:00
Dean Herbert
6268bbcfc8
Merge branch 'master' into beatmap-parsing-fallback-v2
2019-10-03 15:12:21 +08:00
Dean Herbert
d0a4e1e3c2
Force a checksum check before skipping FileStore copy op
2019-09-20 15:00:27 +09:00
Bartłomiej Dach
86588778b1
Implement fallback decoder registration
...
After the preparatory introduction of LineBufferedReader, it is now
possible to introduce registration of fallback decoders that won't drop
input supplied in the first line of the file.
A fallback decoder is used when the magic in the first line of the file
does not match any of the other known decoders. In such a case,
the fallback decoder is constructed and provided a LineBufferedReader
instance. The process of matching magic only peeks the first non-empty
line, so it is available for re-reading in Decode() using ReadLine().
There can be only one fallback decoder per type; a second attempt of
registering a fallback will result in an exception to avoid bugs.
To address the issue of parsing failing on badly or non-headered files,
set the legacy decoders for Beatmaps and Storyboards as the fallbacks.
Due to non-trivial logic, several new, passing unit tests with possible
edge cases also included.
2019-09-15 01:28:07 +02:00
Bartłomiej Dach
7b1ff38df7
Implement line-buffered reader
...
Add a line-buffered reader decorator operating on StreamReader
instances. The decorator has two main operations - PeekLine(), which
allows to see the next line in the stream without consuming it,
ReadLine(), which consumes and returns the next line in the stream, and
ReadToEnd() which reads all the remaining text in the stream (including
the unconsumed peeked line). Peeking line-per-line uses an internal
queue of lines that have been read ahead from the underlying stream.
The addition of the line-buffered reader is a workaround solution to
a problem with decoding. At current selecting a decoder works by
irreversibly reading the first line from the stream and looking for
a magic string that indicates the type of decoder to use.
It might however be possible for a file to be valid in format, just
missing a header. In such a case a lack of a line-buffered reader makes
it impossible to reparse the content of that first line. Introducing it
will however allow to peek the first line for magic first.
- If magic is found in the first line, GetDecoder() will peek it and
use it to return the correct Decoder instance. Note that in the case
of JsonBeatmapDecoder the magic is the opening JSON object brace,
and therefore must not be consumed.
- If magic is not found, the fallback decoder will be able to consume
it using ReadLine() in Decode().
This commit additionally contains basic unit tests for the reader.
Suggested-by: Aergwyn <aergwyn@t-online.de>
2019-09-15 01:26:15 +02:00
Dean Herbert
5b75060b94
Add test for rollback logic correctly dereferencing files
2019-06-10 16:45:45 +09:00
Dean Herbert
80d65f9a3b
Update resource stores with GetAvailableResources
2019-05-31 14:33:18 +09:00
Dean Herbert
612db31c38
Apply newline additions
2019-04-01 12:16:32 +09:00
Dean Herbert
26d53d06a9
Fix remaining issues
2019-02-28 13:31:40 +09:00
Dean Herbert
a580c17949
Fix codefactor issue
2019-01-24 17:46:09 +09:00
Dean Herbert
8617aaa2a7
Update licence header (and remove year)
2019-01-24 17:43:03 +09:00
smoogipoo
a783fdb501
Add LegacyFileArchiveReader
...
Treats files as "archives" for import.
2018-11-28 16:13:16 +09:00
smoogipoo
0982508d26
Rename to LegacyDirectoryArchiveReader
2018-11-28 15:13:27 +09:00
smoogipoo
14ff4bc26a
Move primary key attribute to IHasPrimaryKey
2018-11-28 13:19:23 +09:00
smoogipoo
f27bd3ef3e
OpenTK -> osuTK
2018-11-20 17:14:59 +09:00
smoogipoo
21d5322899
Update with async changes
2018-08-27 17:05:58 +09:00
Dean Herbert
c756a89daf
Standardise path
2018-06-06 20:55:42 +09:00
Dean Herbert
3b03a25ea5
Fix beatmaps with subfolders importing incorrectly on windows
...
Closes #2718 .
2018-06-05 11:36:44 +09:00
smoogipoo
397d93660a
Don't deep-clone beatmapinfo/control points
2018-05-17 13:17:33 +09:00
Dean Herbert
44a9aa529a
Test CI run with new changes
2018-04-22 03:51:17 +09:00
Dean Herbert
32a74f95a5
Normalize all the line endings
2018-04-13 18:26:38 +09:00
smoogipoo
c29f7a4333
Merge remote-tracking branch 'origin/master' into netstandard
2018-03-24 14:49:46 +09:00
Dean Herbert
b9ef32b09b
Further xmldoc and restoring of Cleanup method
2018-02-15 16:31:42 +09:00
Dean Herbert
1b13be1372
Cleanups and xmldoc additions
2018-02-15 16:24:14 +09:00
Dean Herbert
d3dd31dadb
Make deletion and purging logic even more global
2018-02-15 14:22:39 +09:00
Dean Herbert
d340509b1d
Move ArchiveReaders to a more global namespace
...
Also moves delete and action logic to a shared implementation
2018-02-15 14:22:39 +09:00
Dean Herbert
6ff63c2f0c
Move deletion to ArchiveModelImportManager
2018-02-15 14:22:39 +09:00
Dean Herbert
e0d28564d0
Move import logic to shared implementation
2018-02-15 14:22:39 +09:00
Dean Herbert
a738664167
Add interface for database context factory
2018-02-12 23:10:05 +09:00
Dean Herbert
8b37fde15b
Only write when writes occur
...
Also add finaliser logic for safety. Also better threading. Also more cleanup.
2018-02-12 19:57:21 +09:00
Dean Herbert
edc3638175
DatabaseWriteUsage
2018-02-12 17:55:11 +09:00
smoogipoo
b500b76407
Merge branch 'master' into update-branch
2018-02-08 23:35:48 +09:00
Dean Herbert
37d393bca0
Update licence headers
2018-01-05 20:21:19 +09:00
smoogipoo
af86220dcc
Merge remote-tracking branch 'origin/master' into update-branch
2018-01-05 12:08:50 +09:00
smoogipoo
5eb2dc3d64
Cleanup some possible CI errors
...
Not sure why these aren't being detected on master right now - they should be...
2018-01-05 01:20:24 +09:00
smoogipoo
7a793eaded
Merge master into netstandard
2017-12-27 20:48:53 +09:00
smoogipoo
bdf283a4e1
A bit more cleanup
2017-12-07 15:55:38 +09:00
smoogipoo
245b5f759f
Underscore + lowercase all keys
2017-12-07 14:37:12 +09:00
smoogipoo
d026587a91
Add flag to explicitly serialize the type version
2017-12-07 12:49:16 +09:00
smoogipoo
9e51480aa3
Cleanup TypedListConverter a bit
2017-12-07 12:27:30 +09:00
smoogipoo
feef4b1890
Add license header
2017-12-07 12:18:00 +09:00
smoogipoo
851c20aff0
Add a few comments
2017-12-07 11:17:32 +09:00
smoogipoo
9787788081
Revert unintended change
2017-12-07 03:39:43 +09:00
smoogipoo
4232a54b32
Make TypedListConverter not reconstruct serializers
2017-12-07 03:04:56 +09:00
smoogipoo
b6b26cfe25
Add basic method to migrate beatmaps to the new JSON format
2017-12-07 03:04:36 +09:00
smoogipoo
e199ee5225
Add a few xmldocs
2017-12-06 00:42:17 +09:00
smoogipoo
b584178e85
Make Beatmap ISerializable and add more JsonIgnores
2017-12-06 00:37:37 +09:00
smoogipoo
aac41d2de6
Disable resharper inspections on case-by-case basis
2017-11-20 18:55:48 +09:00
smoogipoo
d2ab0621f3
Make osu.Game + rulesets compile with netstandard
2017-11-19 13:46:51 +09:00
smoogipoo
04b271cce9
Migrate osu.Game project to VS2017 project structure
2017-10-30 22:24:11 +09:00
TocoToucan
5107489cda
Remove DatabaseBackedStore Reset functions
2017-10-25 15:12:14 +03:00
TocoToucan
63fe569afa
Remove DatabaseBackedStore Prepare functions
2017-10-25 14:53:20 +03:00
Dean Herbert
df20845fbb
Share one context per file store for performance reasons
...
There is now a CreateContext method for retrieving a stand-alone context for threaded use.
We may want to add safety against this context being disposed (or just return a fresh one if it is).
2017-10-23 16:35:35 +09:00
Dean Herbert
4841d4a937
Fix deletion and use single context for imports and deletions for now
2017-10-18 18:27:17 +09:00
Dean Herbert
e4a066dc5f
Run cleanup tasks only on startup via manual calls
2017-10-17 19:58:33 +09:00
Dean Herbert
64dfce258f
Fix file prefix not being read when calling storage.Exists
2017-10-17 17:08:19 +09:00
Dean Herbert
e487b6f82a
Standardise context retrieval
2017-10-17 15:50:42 +09:00
Dean Herbert
cd41862e3b
Add back transaction support for beatmap importing
2017-10-17 15:00:27 +09:00
Dean Herbert
66894d11ea
Connection -> context
2017-10-17 12:46:29 +09:00
Dean Herbert
0df474accb
Simplify file storing
2017-10-17 11:22:50 +09:00
Dean Herbert
31dc5c97f2
Fix intro and duplicate inserts
2017-10-17 11:22:42 +09:00
Dean Herbert
799f510217
FileStore logic fixes
2017-10-17 10:25:18 +09:00
Dean Herbert
7049a73490
Remove ValidTypes from databased stored; explicitly expose query methods instead
2017-10-16 14:47:33 +09:00
Dean Herbert
0a6dcdd405
Fix remaining warnings
2017-10-16 13:09:10 +09:00
smoogipoo
b178be21d1
Let's not rename columns yet
2017-10-14 14:30:58 +09:00
TocoToucan
4653470779
Fix RulesetStore preparation
2017-10-10 22:29:16 +03:00
TocoToucan
ef417c6388
Small DB-related refactoring
2017-10-09 23:30:32 +03:00
TocoToucan
887aa7496b
Rework EF Core usage in model classes
2017-10-06 00:23:26 +03:00
TocoToucan
6a4198d0d6
Initial EF Core commit
2017-10-04 22:52:12 +03:00
Huo Yaoyuan
87c8278139
Use Array.Empty.
2017-09-24 03:45:46 +08:00
Dean Herbert
c41ca10715
Allow files missing on disk to be restored on beatmap import
...
Previously, in the rare case the database became out of sync with the disk store, it was impossible to feasibly repair a beatmap. Now reimporting checks each file exists on disk and adds it back if it doesn't.
2017-09-19 18:35:52 +09:00
Dean Herbert
e75f438c29
Add difficulty application mods
...
Also fixes circular references when using IJsonSerializable.
2017-08-07 08:54:54 +09:00
Dean Herbert
6eb960010f
Speed up startup cleanup operations.
2017-08-01 17:58:21 +09:00
Dean Herbert
3b1166d1e6
Optimise file lookups and other database operations
...
FirstOrDefault when called on a TableQuery with a predicate doesn't use table indices
2017-08-01 17:37:21 +09:00
Dean Herbert
9d630e446e
Use new storage methods to reset FileStore
...
Guarantees that backing files are cleaned up correctly. Also handles lingering "beatmaps" directory from older builds.
2017-08-01 10:57:46 +09:00
Dean Herbert
821f65c381
Actually delete files
2017-08-01 10:24:12 +09:00
Dean Herbert
c73e139954
Add "migration"
...
Also simplify initial migration for BeatmapStore by just nuking everything.
2017-08-01 10:24:04 +09:00
Dean Herbert
c060d32765
Separate out startup tasks to ensure they run after migrations
2017-08-01 10:24:00 +09:00
Dean Herbert
404497fa10
Allow a single beatmap to reference the same file multiple times
...
This fixes incorrect reference counts causing database desync.
2017-07-31 21:49:57 +09:00
Dean Herbert
6f4efbeda6
Merge remote-tracking branch 'upstream/master' into database-tidy
2017-07-27 17:41:29 +09:00
Dean Herbert
996e5e8b4a
Use SHA2 for hashing
2017-07-27 17:40:08 +09:00
Dean Herbert
5f53426a9a
*Database -> *Store
...
Welcome back BeatmapManager
2017-07-27 16:56:41 +09:00
Dean Herbert
96b08b8777
Simplify and document DatabaseStore API
2017-07-27 15:06:10 +09:00
Dean Herbert
898a601098
Introduce a reference counting file store
2017-07-26 20:22:02 +09:00
Huo Yaoyuan
e2a7f00a52
Remove unused namespace.
2017-05-16 22:20:05 +08:00
Huo Yaoyuan
e911441394
Why AbandonedMutexException?
2017-05-16 21:23:14 +08:00
Huo Yaoyuan
e466380ee2
Generic infer for IJsonSerializable.DeepClone.
2017-05-08 19:33:37 +08:00
smoogipooo
cec8bca78a
Implement IJsonSerializable, BeatmapInfo IJsonSerializable.
2017-04-06 15:54:50 +09:00
Dean Herbert
54e1b24fe9
Enforce readonly private members where possible.
2017-03-23 13:50:08 +09:00
Dean Herbert
ba10c3a8db
Revert "Merge pull request #441 from tacchinotacchi/archive-reader"
...
This reverts commit 6f20473e65c76a495d3d
2017-03-14 11:46:34 +09:00
Alex Amadori
c84a9d56f5
Fix namespace, comply with naming rules
2017-03-13 11:36:21 +01:00
Alex Amadori
caa6e9c82b
General improvements
2017-03-13 11:29:24 +01:00
Dean Herbert
d7497330b2
Reduce warning/suggestion count to zero.
2017-03-09 15:52:40 +09:00
Dean Herbert
da751804b6
Reduce error count.
2017-03-09 14:24:16 +09:00
Dean Herbert
dc5ac6e88f
Add missing access modifiers.
2017-03-09 13:55:29 +09:00
Dean Herbert
5a42035f88
Update license headers.
2017-03-08 15:50:52 +09:00
Dean Herbert
58ae9e888d
Basic partial replay support.
2017-03-01 22:04:20 +09:00
