h1or4dme8/osu-lazer
1
0
Fork 0
mirror of https://github.com/ppy/osu.git synced 2024-12-21 18:02:55 +08:00
Code Issues Releases Wiki Activity
53,222 Commits 5 Branches 664 Tags 357 MiB
e58c365276
Commit Graph

231 Commits

Author SHA1 Message Date
Dean Herbert
0690d81bbb Add protection against migrating to a nested folder 2020-05-15 10:40:26 +09:00
Dean Herbert
25bbb02999 Throw better exceptions from OsuStorage 2020-05-15 10:28:03 +09:00
Dean Herbert
364aa5aa12 Add protection against migrating to a nested folder 2020-05-14 22:45:03 +09:00
Dean Herbert
827d75b152 Revert "Add protection against migrating to a nested folder" 
This reverts commit ef8375b442.
 2020-05-14 22:44:27 +09:00
Dean Herbert
ef8375b442 Add protection against migrating to a nested folder 2020-05-14 22:42:42 +09:00
Dean Herbert
06f507496a Delete migration source if no files exist after completion 2020-05-14 17:41:55 +09:00
Dean Herbert
cb0b25ac55 Throw better exceptions from OsuStorage 2020-05-14 17:41:55 +09:00
Dean Herbert
ad1d050fb4 Throw exception on copy timeout 2020-05-13 20:29:15 +09:00
Dean Herbert
49e616b7e5 Also check for directory presence before migrating 2020-05-13 20:19:14 +09:00
Dean Herbert
9de216f1b4 Merge branch 'master' into migration-backend 2020-05-13 20:15:22 +09:00
Huo Yaoyuan
f07d95ac59 Use IReadOnlyList<T> for TypedListConverter. 2020-05-12 21:49:55 +08:00
Huo Yaoyuan
f7d26688f4 Merge branch 'master' into json-converter 2020-05-12 21:43:02 +08:00
Dean Herbert
e4a23b3e7d Fix ignored excluding more than top level 2020-05-12 12:39:04 +09:00
Dean Herbert
984f27c107 Add simple retry logic on file copy failure (may be in use) 2020-05-11 21:39:44 +09:00
Dean Herbert
a11be07bb1 Move log storage location after migration complete 2020-05-11 21:39:44 +09:00
Dean Herbert
36640f5dda Merge branch 'master' into migration-backend 2020-05-11 18:52:36 +09:00
Dean Herbert
384862d48b Fix incorrect relative paths when using GetStorageForDirectory 2020-05-10 13:20:13 +09:00
Dean Herbert
bbebd26efb Use DirectoryInfo in more places 2020-05-09 20:13:31 +09:00
Huo Yaoyuan
80a193a616 Use IEnumerable<T> for TypedListConverter. 2020-05-09 17:41:59 +08:00
Huo Yaoyuan
74cbe9306c Use strongly-typed JsonConerter. 2020-05-09 16:39:46 +08:00
Dean Herbert
7a2020fd45 User copy operation instead of move 2020-05-07 22:12:24 +09:00
Dean Herbert
49a03f1c06 Add basic blocking migration, move not copy 2020-05-07 22:11:57 +09:00
Dean Herbert
7781408643 Update in line with framework storage changes 2020-05-06 18:28:24 +09:00
smoogipoo
d27ca725f9 Use IEnumerable<T> instead 2020-04-22 09:04:44 +09:00
smoogipoo
a702a521f8 Fix not being able to serialise converted beatmaps 2020-04-22 09:04:44 +09:00
smoogipoo
f6f5de7ad1 Allow LineBufferedReader to keep underlying stream open 2020-03-31 10:13:50 +09:00
Dean Herbert
022465f546 Add encoding and import support 2020-03-24 14:51:52 +09:00
Dean Herbert
68ebe98fde Remove unused GetUnderlyingStream method 2020-03-24 14:08:25 +09:00
Huo Yaoyuan
b14c7bd171 Use type switch in SearializationWriter. 2020-01-27 21:49:34 +08:00
Huo Yaoyuan
40b43b85f1 CA1715: use prefix for generic parameters. 2019-12-17 13:00:25 +08:00
Huo Yaoyuan
e46f6627e4 CA1052: make type static. 2019-12-17 13:00:13 +08:00
Huo Yaoyuan
31cc0d13da Use 'out var'. 2019-11-12 19:55:26 +08:00
Dean Herbert
35be8f9dfb Share framework file-exclusion function 2019-10-30 19:34:17 +09:00
Bartłomiej Dach
4b84564f47
Switch casing comparison mode to ordinal 
Switch from InvariantCultureIgnoreCase to OrdinalIgnoreCase when
checking file paths in archives for substrings indicating the file can
be ignored for performance gains.

Co-Authored-By: Dan Balasescu <smoogipoo@smgi.me>
 2019-10-11 09:24:41 +02:00
Bartłomiej Dach
57bfa18359 Filter out OS-generated files from archives 
Add a filename ignore list to ZipArchiveReader to filter out superfluous
OS-generated files from archives during the import process. In addition
to decreasing the size of files imported this allows imports of some
incorrectly-constructed archives. An example is the case of having
a __MACOSX directory next to a single directory with the actual files -
filtering out the former at ZipArchiveReader allows the fallback added
in #6170 to work.
 2019-10-10 23:54:23 +02:00
Dean Herbert
6268bbcfc8
Merge branch 'master' into beatmap-parsing-fallback-v2 2019-10-03 15:12:21 +08:00
Dean Herbert
d0a4e1e3c2 Force a checksum check before skipping FileStore copy op 2019-09-20 15:00:27 +09:00
Bartłomiej Dach
86588778b1 Implement fallback decoder registration 
After the preparatory introduction of LineBufferedReader, it is now
possible to introduce registration of fallback decoders that won't drop
input supplied in the first line of the file.

A fallback decoder is used when the magic in the first line of the file
does not match any of the other known decoders. In such a case,
the fallback decoder is constructed and provided a LineBufferedReader
instance. The process of matching magic only peeks the first non-empty
line, so it is available for re-reading in Decode() using ReadLine().

There can be only one fallback decoder per type; a second attempt of
registering a fallback will result in an exception to avoid bugs.

To address the issue of parsing failing on badly or non-headered files,
set the legacy decoders for Beatmaps and Storyboards as the fallbacks.

Due to non-trivial logic, several new, passing unit tests with possible
edge cases also included.
 2019-09-15 01:28:07 +02:00
Bartłomiej Dach
7b1ff38df7 Implement line-buffered reader 
Add a line-buffered reader decorator operating on StreamReader
instances. The decorator has two main operations - PeekLine(), which
allows to see the next line in the stream without consuming it,
ReadLine(), which consumes and returns the next line in the stream, and
ReadToEnd() which reads all the remaining text in the stream (including
the unconsumed peeked line). Peeking line-per-line uses an internal
queue of lines that have been read ahead from the underlying stream.

The addition of the line-buffered reader is a workaround solution to
a problem with decoding. At current selecting a decoder works by
irreversibly reading the first line from the stream and looking for
a magic string that indicates the type of decoder to use.

It might however be possible for a file to be valid in format, just
missing a header. In such a case a lack of a line-buffered reader makes
it impossible to reparse the content of that first line. Introducing it
will however allow to peek the first line for magic first.

 - If magic is found in the first line, GetDecoder() will peek it and
   use it to return the correct Decoder instance. Note that in the case
   of JsonBeatmapDecoder the magic is the opening JSON object brace,
   and therefore must not be consumed.

 - If magic is not found, the fallback decoder will be able to consume
   it using ReadLine() in Decode().

This commit additionally contains basic unit tests for the reader.

Suggested-by: Aergwyn <aergwyn@t-online.de>
 2019-09-15 01:26:15 +02:00
Dean Herbert
5b75060b94 Add test for rollback logic correctly dereferencing files 2019-06-10 16:45:45 +09:00
Dean Herbert
80d65f9a3b Update resource stores with GetAvailableResources 2019-05-31 14:33:18 +09:00
Dean Herbert
612db31c38 Apply newline additions 2019-04-01 12:16:32 +09:00
Dean Herbert
26d53d06a9 Fix remaining issues 2019-02-28 13:31:40 +09:00
Dean Herbert
a580c17949 Fix codefactor issue 2019-01-24 17:46:09 +09:00
Dean Herbert
8617aaa2a7 Update licence header (and remove year) 2019-01-24 17:43:03 +09:00
smoogipoo
a783fdb501 Add LegacyFileArchiveReader 
Treats files as "archives" for import.
 2018-11-28 16:13:16 +09:00
smoogipoo
0982508d26 Rename to LegacyDirectoryArchiveReader 2018-11-28 15:13:27 +09:00
smoogipoo
14ff4bc26a Move primary key attribute to IHasPrimaryKey 2018-11-28 13:19:23 +09:00
smoogipoo
f27bd3ef3e OpenTK -> osuTK 2018-11-20 17:14:59 +09:00
smoogipoo
21d5322899 Update with async changes 2018-08-27 17:05:58 +09:00
Dean Herbert
c756a89daf Standardise path 2018-06-06 20:55:42 +09:00
Dean Herbert
3b03a25ea5 Fix beatmaps with subfolders importing incorrectly on windows 
Closes #2718.
 2018-06-05 11:36:44 +09:00
smoogipoo
397d93660a Don't deep-clone beatmapinfo/control points 2018-05-17 13:17:33 +09:00
Dean Herbert
44a9aa529a Test CI run with new changes 2018-04-22 03:51:17 +09:00
Dean Herbert
32a74f95a5 Normalize all the line endings 2018-04-13 18:26:38 +09:00
smoogipoo
c29f7a4333 Merge remote-tracking branch 'origin/master' into netstandard 2018-03-24 14:49:46 +09:00
Dean Herbert
b9ef32b09b Further xmldoc and restoring of Cleanup method 2018-02-15 16:31:42 +09:00
Dean Herbert
1b13be1372 Cleanups and xmldoc additions 2018-02-15 16:24:14 +09:00
Dean Herbert
d3dd31dadb Make deletion and purging logic even more global 2018-02-15 14:22:39 +09:00
Dean Herbert
d340509b1d Move ArchiveReaders to a more global namespace 
Also moves delete and action logic to a shared implementation
 2018-02-15 14:22:39 +09:00
Dean Herbert
6ff63c2f0c Move deletion to ArchiveModelImportManager 2018-02-15 14:22:39 +09:00
Dean Herbert
e0d28564d0 Move import logic to shared implementation 2018-02-15 14:22:39 +09:00
Dean Herbert
a738664167 Add interface for database context factory 2018-02-12 23:10:05 +09:00
Dean Herbert
8b37fde15b Only write when writes occur 
Also add finaliser logic for safety. Also better threading. Also more cleanup.
 2018-02-12 19:57:21 +09:00
Dean Herbert
edc3638175 DatabaseWriteUsage 2018-02-12 17:55:11 +09:00
smoogipoo
b500b76407 Merge branch 'master' into update-branch 2018-02-08 23:35:48 +09:00
Dean Herbert
37d393bca0 Update licence headers 2018-01-05 20:21:19 +09:00
smoogipoo
af86220dcc Merge remote-tracking branch 'origin/master' into update-branch 2018-01-05 12:08:50 +09:00
smoogipoo
5eb2dc3d64 Cleanup some possible CI errors 
Not sure why these aren't being detected on master right now - they should be...
 2018-01-05 01:20:24 +09:00
smoogipoo
7a793eaded Merge master into netstandard 2017-12-27 20:48:53 +09:00
smoogipoo
bdf283a4e1 A bit more cleanup 2017-12-07 15:55:38 +09:00
smoogipoo
245b5f759f Underscore + lowercase all keys 2017-12-07 14:37:12 +09:00
smoogipoo
d026587a91 Add flag to explicitly serialize the type version 2017-12-07 12:49:16 +09:00
smoogipoo
9e51480aa3 Cleanup TypedListConverter a bit 2017-12-07 12:27:30 +09:00
smoogipoo
feef4b1890 Add license header 2017-12-07 12:18:00 +09:00
smoogipoo
851c20aff0 Add a few comments 2017-12-07 11:17:32 +09:00
smoogipoo
9787788081 Revert unintended change 2017-12-07 03:39:43 +09:00
smoogipoo
4232a54b32 Make TypedListConverter not reconstruct serializers 2017-12-07 03:04:56 +09:00
smoogipoo
b6b26cfe25 Add basic method to migrate beatmaps to the new JSON format 2017-12-07 03:04:36 +09:00
smoogipoo
e199ee5225 Add a few xmldocs 2017-12-06 00:42:17 +09:00
smoogipoo
b584178e85 Make Beatmap ISerializable and add more JsonIgnores 2017-12-06 00:37:37 +09:00
smoogipoo
aac41d2de6 Disable resharper inspections on case-by-case basis 2017-11-20 18:55:48 +09:00
smoogipoo
d2ab0621f3 Make osu.Game + rulesets compile with netstandard 2017-11-19 13:46:51 +09:00
smoogipoo
04b271cce9 Migrate osu.Game project to VS2017 project structure 2017-10-30 22:24:11 +09:00
TocoToucan
5107489cda Remove DatabaseBackedStore Reset functions 2017-10-25 15:12:14 +03:00
TocoToucan
63fe569afa Remove DatabaseBackedStore Prepare functions 2017-10-25 14:53:20 +03:00
Dean Herbert
df20845fbb Share one context per file store for performance reasons 
There is now a CreateContext method for retrieving a stand-alone context for threaded use.

We may want to add safety against this context being disposed (or just return a fresh one if it is).
 2017-10-23 16:35:35 +09:00
Dean Herbert
4841d4a937 Fix deletion and use single context for imports and deletions for now 2017-10-18 18:27:17 +09:00
Dean Herbert
e4a066dc5f Run cleanup tasks only on startup via manual calls 2017-10-17 19:58:33 +09:00
Dean Herbert
64dfce258f Fix file prefix not being read when calling storage.Exists 2017-10-17 17:08:19 +09:00
Dean Herbert
e487b6f82a Standardise context retrieval 2017-10-17 15:50:42 +09:00
Dean Herbert
cd41862e3b Add back transaction support for beatmap importing 2017-10-17 15:00:27 +09:00
Dean Herbert
66894d11ea Connection -> context 2017-10-17 12:46:29 +09:00
Dean Herbert
0df474accb Simplify file storing 2017-10-17 11:22:50 +09:00
Dean Herbert
31dc5c97f2 Fix intro and duplicate inserts 2017-10-17 11:22:42 +09:00
Dean Herbert
799f510217 FileStore logic fixes 2017-10-17 10:25:18 +09:00
Dean Herbert
7049a73490 Remove ValidTypes from databased stored; explicitly expose query methods instead 2017-10-16 14:47:33 +09:00
Dean Herbert
0a6dcdd405 Fix remaining warnings 2017-10-16 13:09:10 +09:00
smoogipoo
b178be21d1 Let's not rename columns yet 2017-10-14 14:30:58 +09:00
TocoToucan
4653470779 Fix RulesetStore preparation 2017-10-10 22:29:16 +03:00
TocoToucan
ef417c6388 Small DB-related refactoring 2017-10-09 23:30:32 +03:00
TocoToucan
887aa7496b Rework EF Core usage in model classes 2017-10-06 00:23:26 +03:00
TocoToucan
6a4198d0d6 Initial EF Core commit 2017-10-04 22:52:12 +03:00
Huo Yaoyuan
87c8278139 Use Array.Empty. 2017-09-24 03:45:46 +08:00
Dean Herbert
c41ca10715 Allow files missing on disk to be restored on beatmap import 
Previously, in the rare case the database became out of sync with the disk store, it was impossible to feasibly repair a beatmap. Now reimporting checks each file exists on disk and adds it back if it doesn't.
 2017-09-19 18:35:52 +09:00
Dean Herbert
e75f438c29 Add difficulty application mods 
Also fixes circular references when using IJsonSerializable.
 2017-08-07 08:54:54 +09:00
Dean Herbert
6eb960010f Speed up startup cleanup operations. 2017-08-01 17:58:21 +09:00
Dean Herbert
3b1166d1e6 Optimise file lookups and other database operations 
FirstOrDefault when called on a TableQuery with a predicate doesn't use table indices
 2017-08-01 17:37:21 +09:00
Dean Herbert
9d630e446e Use new storage methods to reset FileStore 
Guarantees that backing files are cleaned up correctly. Also handles lingering "beatmaps" directory from older builds.
 2017-08-01 10:57:46 +09:00
Dean Herbert
821f65c381 Actually delete files 2017-08-01 10:24:12 +09:00
Dean Herbert
c73e139954 Add "migration" 
Also simplify initial migration for BeatmapStore by just nuking everything.
 2017-08-01 10:24:04 +09:00
Dean Herbert
c060d32765 Separate out startup tasks to ensure they run after migrations 2017-08-01 10:24:00 +09:00
Dean Herbert
404497fa10 Allow a single beatmap to reference the same file multiple times 
This fixes incorrect reference counts causing database desync.
 2017-07-31 21:49:57 +09:00
Dean Herbert
6f4efbeda6 Merge remote-tracking branch 'upstream/master' into database-tidy 2017-07-27 17:41:29 +09:00
Dean Herbert
996e5e8b4a Use SHA2 for hashing 2017-07-27 17:40:08 +09:00
Dean Herbert
5f53426a9a *Database -> *Store 
Welcome back BeatmapManager
 2017-07-27 16:56:41 +09:00
Dean Herbert
96b08b8777 Simplify and document DatabaseStore API 2017-07-27 15:06:10 +09:00
Dean Herbert
898a601098 Introduce a reference counting file store 2017-07-26 20:22:02 +09:00
Huo Yaoyuan
e2a7f00a52 Remove unused namespace. 2017-05-16 22:20:05 +08:00
Huo Yaoyuan
e911441394 Why AbandonedMutexException? 2017-05-16 21:23:14 +08:00
Huo Yaoyuan
e466380ee2 Generic infer for IJsonSerializable.DeepClone. 2017-05-08 19:33:37 +08:00
smoogipooo
cec8bca78a Implement IJsonSerializable, BeatmapInfo IJsonSerializable. 2017-04-06 15:54:50 +09:00
Dean Herbert
54e1b24fe9
Enforce readonly private members where possible. 2017-03-23 13:50:08 +09:00
Dean Herbert
ba10c3a8db
Revert "Merge pull request #441 from tacchinotacchi/archive-reader" 
This reverts commit 6f20473e65, reversing
changes made to c76a495d3d.
 2017-03-14 11:46:34 +09:00
Alex Amadori
c84a9d56f5 Fix namespace, comply with naming rules 2017-03-13 11:36:21 +01:00
Alex Amadori
caa6e9c82b General improvements 2017-03-13 11:29:24 +01:00
Dean Herbert
d7497330b2
Reduce warning/suggestion count to zero. 2017-03-09 15:52:40 +09:00
Dean Herbert
da751804b6
Reduce error count. 2017-03-09 14:24:16 +09:00
Dean Herbert
dc5ac6e88f
Add missing access modifiers. 2017-03-09 13:55:29 +09:00
Dean Herbert
5a42035f88
Update license headers. 2017-03-08 15:50:52 +09:00
Dean Herbert
58ae9e888d
Basic partial replay support. 2017-03-01 22:04:20 +09:00