e17a54ba3c
feat: advancing on new script
main
Leonard Excoffier
2024-09-10 22:23:14 -04:00
e586c2b202
feat: downloads and extract the zip.
Leonard Excoffier
2024-09-10 21:38:43 -04:00
5e38f46e8f
Temp expirimentation, problem with duplicate primary key lowercase uppercase and insert on duplicate, need to find way to bulk insert but still not insert rows that have duplicate.
Leonard Excoffier
2024-08-31 22:34:04 -04:00
162106c8e0
e
Leonard Excoffier
2024-08-31 19:50:15 -04:00
fa64f81cb8
clean slate
Leonard Excoffier
2024-08-31 19:14:02 -04:00
91272506e3
feat: drops columns that are not supposed to be there
Leonard Excoffier
2024-08-31 18:54:28 -04:00
3f5f6d5569
fix: remove not null in some places
Leonard Excoffier
2024-08-31 18:26:54 -04:00
84b20e1496
fix: removed NOT NULL specification for non primary keys
Leonard Excoffier
2024-08-31 18:26:14 -04:00
4c9b2e46c1
fix: put the doc type back to TEXT
Leonard Excoffier
2024-08-31 18:23:48 -04:00
c9612b35ae
fix: added some special fixe that are not taken into account by dataset documentation
Leonard Excoffier
2024-08-31 17:18:33 -04:00
025ba9b913
fix: fixed coreg being a null value issue
Leonard Excoffier
2024-08-31 17:17:17 -04:00
57b13dbec5
feat: based on the sec doc match the sql schema
Leonard Excoffier
2024-08-31 16:02:39 -04:00
00a59b27ce
feat: now writes all datasets to db
Leonard Excoffier
2024-08-31 15:52:37 -04:00
015879ef4d
feat: added reset block
Leonard Excoffier
2024-08-31 15:45:53 -04:00
3a60674b7c
feaet: now cycles through wanted files
Leonard Excoffier
2024-08-31 15:41:33 -04:00
f31d901201
refactor: total change, no longer ripping gigabytes but going directly to datasets, must find way to automate download.
Leonard Excoffier
2024-08-31 15:32:57 -04:00
b5e0af3ca5
feat: added some helpfull commands
Leonard Excoffier
2024-08-31 13:00:04 -04:00
ab8f1fcf8f
feat: now use batches but seem to write less rows, not sure
Leonard Excoffier
2024-08-31 00:50:17 -04:00
9c5afa7670
feat: script cycle works
Leonard Excoffier
2024-08-31 00:38:50 -04:00
765272f896
feat: seem to be able to write to a db.
Leonard Excoffier
2024-08-31 00:17:37 -04:00
0fbbb37ef7
feat: design new schema
Leonard Excoffier
2024-08-30 19:59:42 -04:00
71a054e2b6
feat: new script seem to be super efficient
Leonard Excoffier
2024-08-30 19:54:37 -04:00
6fcbc73a4f
fix: correctly adapted script for new primary key
Leonard Excoffier
2024-08-30 08:16:05 -04:00
184f82a688
fix: correctly set primary key for data table
Leonard Excoffier
2024-08-30 07:56:02 -04:00
0f9a42eb7e
feat: now checks if cik is already present before writing to db, can be deactivated if want to update data.
Leonard Excoffier
2024-08-29 22:04:47 -04:00
a17d73f336
feat: now loops over all the facts jsons
Leonard Excoffier
2024-08-29 22:03:05 -04:00
5946ff73bd
feat: script to write info to db seems to work, have no idea how tho, need to generalize with a for loop for all files
Leonard Excoffier
2024-08-29 21:51:05 -04:00
750d33b976
feat: added stuff and corrected the schema creation.
Leonard Excoffier
2024-08-29 21:39:57 -04:00
d5d823b290
feat: changed primary key of facts table
Leonard Excoffier
2024-08-29 21:35:30 -04:00
97bd73c4a3
fix: stopped download and extraction of submissions, only keep company facts
Leonard Excoffier
2024-08-27 22:11:27 -04:00
4266e09691
feat: script to output all the labels , initiation to parsing jsons, may use jquery on mac.
Leonard Excoffier
2024-08-27 21:58:16 -04:00
ce0152bbba
feat: completed db schema
Leonard Excoffier
2024-08-27 19:13:07 -04:00
19a5579fb0
feat: db schema has now a base
Leonard Excoffier
2024-08-27 19:09:12 -04:00
c02fea1417
feat: initiated description of db schema based on facts json.
Leonard Excoffier
2024-08-26 22:08:28 -04:00
20a2866ecd
batman
Leonard Excoffier
2024-08-25 19:52:34 -04:00
Leonard Excoffier