The OpenNET Project / Index page

[ новости /+++ | форум | теги | ]

форумы
правила/FAQ
поиск
регистрация
вход
слежка
RSS


Вариант для распечатки  
Пред. тема | След. тема 
Форум Разговоры, обсуждение новостей
Режим отображения отдельной подветви беседы [ Отслеживать ]

Оглавление

Эксперимент с использованием SQLite в качестве контейнера для архивирования файлов, opennews (??), 25-Мрт-24, (0) [смотреть все]

Сообщения [Сортировка по времени | RSS]


63. "Эксперимент с использованием SQLite в качестве контейнера дл..."  +/–
Сообщение от Аноним (84), 25-Мрт-24, 21:23 
>SQLite security:
>It is one, if not the most secure, library out there. It is very hard to crack it, and it will not allow running any harmful code on a machine. It is used in almost anything with a computer, partially because of its security and reliability.

https://www.blackhat.com/docs/us-17/wednesday/us-17-Feng-Man...

Yes, I have read https://www.sqlite.org/security.html , but
* I still don't believe it is possible to make SQLite secure as an exchange format, there is a long trail of vulnrs in it allowing to achieve an RCE triggered by just opening a maliciously crafted database file and SELECTing from it.
* IMHO quality metrics for a good RDBMS are different from the ones of a good archiver. Everything is a tradeoff and there is ain't no such a thing as free lunch. RDBMS are information retrieval tools, they require good performance on wide ranges of queries and are usually operated on trusted data, so sacrificing some amount of security for performance is a tradeoff good RDBMS have to make. Archivers also need to be performant, but they are almost always operated on files from untrusted sources (downloaded from the Internet from random web pages) and so first of all they need to be secure, and then queries for them are pretty limited (basically it is a key-value storage), so a good archiver should optimize storage format for that purpose.

I know that when one has a hammer, all problems look like a nail, but let's drive nails with hammers, not with microscopes ("drive nails with microscopes" is a Russian idiom, I hope you get its meaning).

Ответить | Правка | Наверх | Cообщить модератору

65. "Эксперимент с использованием SQLite в качестве контейнера дл..."  +/–
Сообщение от Аноним (84), 25-Мрт-24, 21:32 
And forgot to add:
* For the purpose of this paragraph let's assumme that SQLite when built with maximum hardening is secure ... Even if you use maximum source-level hardening options of SQLite, it would limit your software to 2 options: statically linking the SQLite lib with thisenoptions, or dynamically linking an additional variant of SQLite lib living in a separate package. Some distro maintainers can forget about this security measure and just link the usual distrowise SQLite lib optimized for performance and special tasks like schema manipulation ... creating this way a vulnerability. So mere using SQLite for the task it is unsuitable for introduces a weak point.
Ответить | Правка | Наверх | Cообщить модератору

80. "Эксперимент с использованием SQLite в качестве контейнера дл..."  +/–
Сообщение от O (?), 26-Мрт-24, 00:55 
All are an will be statically linked. They are not considered a shared library to the program, but part of the source.
Ответить | Правка | Наверх | Cообщить модератору

82. "Эксперимент с использованием SQLite в качестве контейнера дл..."  –3 +/–
Сообщение от Аноним (84), 26-Мрт-24, 02:21 
Noone sane uses statically-linked libs and sane distros would never accept anything using statically linked libs. It is unmaintainable shit. If a yet another critical RCE vulnr is found in your precious SQLite, then the lib in the distro will be upgraded, but your archiver (needed by nobody sane and kept only to make a check mark that they have it in the repo, if it got enough adoption) with statically linked SQLite will stay vulnerable.

We have enough pain in the ass with Python's pickle, which should be considered a backdoor. If your archiver gets any adoption, there will be a yet another backdoor. Yes, I consider your archive format as a backdoor, and the attempt to forcibly promote it, to the point you are tracking mentions of it on websites in foreign languages, as an attempt to promote a hard-to-remove (if it got adoption and there would be enough archives of your format with valuable unique content in the Internet, so users would have no other option, but to either use SQLite and tolerate the risk, or create an own impl of SQLite specifucally designed to deal with malicious databases securely) backdoor.

Ответить | Правка | Наверх | Cообщить модератору

87. "Эксперимент с использованием SQLite в качестве контейнера дл..."  +/–
Сообщение от Аноним (62), 26-Мрт-24, 10:43 
Ты просто не пользовался скулайтом, он примерно всегда бандлится. Или не заметил. Ну и все мейнтейнеры жрут, что им навалят разрабы, странный аргумент. Потому что шляпа вроде ffmpeg или libvpx регулярно ломает совместимость, но узнаешь ты об этом только когда пользователи начнут ныть (подход компилируется значит работает очень популярный у мейнтенеров). Или другой пример именно статически линкуемого компонента неизвестной версии это zlib.
Ответить | Правка | Наверх | Cообщить модератору

89. "Эксперимент с использованием SQLite в качестве контейнера дл..."  +/–
Сообщение от Аноним (89), 26-Мрт-24, 13:07 
>Ты просто не пользовался скулайтом, он примерно всегда бандлится

У неадекватов всё подряд бандлится, причём с обоснованием уровня "а вот я тут главный, хочу так, и ниипёт". Некоторые доходят до того, чтш используют Rust или поставляют свои программы в формате Snap или Docker-контейнеров.

zlib и sqlite вообще отличаются довольно стабильным API и очень широко используются, я не помню, чтобы хоть раз какой-либо софт сломался из-за этих либ, прилинкованных динамически.

Вот пример адекватного пакетирования
https://packages.debian.org/sid/python3.12-minimal , https://packages.debian.org/sid/libzip4t64 , https://packages.debian.org/sid/libarchive13t64 , https://packages.debian.org/ru/sid/libxft-dev , https://packages.debian.org/buster/libsvn-dev (остальное найдёшь сам, меня забанили в Гугле. zlib1g AND -inurl:zlib1g AND -inurl:lib32z1 AND -inurl:lib64z1 AND -inurl:search AND -inurl:search_contents AND -inurl:zlib AND site:packages.debian.org) - зависит от https://packages.debian.org/sid/zlib1g
https://packages.debian.org/sid/libpython3.12-stdlib - зависит от https://packages.debian.org/sid/libsqlite3-0

А свои сказки про то, что всегда бандлится ... я уже сказал, бандлится только у неадекватов, которым лень разобраться в работе CMake, поэтому у которых всё на git-подмодулях или вообще просто скопировано в дерево исходников.

Ответить | Правка | Наверх | Cообщить модератору

93. "Эксперимент с использованием SQLite в качестве контейнера дл..."  +/–
Сообщение от Аноним (62), 26-Мрт-24, 13:24 
Дело не в этом. Лучше смотри на это с позиции когда мейнтенеры не будут возиться с разбандливанием и разгребанием багов в каждой поделке (а они там будут). Если говорить о скулите, то дистрибутивная версия может быть собрана без secure-delete (потому что это угрожает производительности), а та, что поставляется, например, с браузером, компилируется с этим флагом. Версии, поставляемые с браузером,  во многих случаях будут более новые, либо с применёнными (иными) патчами. В целом, особенно актуальны вопросы совместимости для программ на плюсах, сишные в значительной мере совместимы. Но всё равно нельзя взять и подсунуть произвольную версию и только разрабы знают какая подходит и почему, не мейнтейнеры. Куда чаще проблема не в том, что разрабы не разбираются, а в том, что мейнтейнеры разбираются недостаточно хорошо.
Ответить | Правка | Наверх | Cообщить модератору

96. "Эксперимент с использованием SQLite в качестве контейнера дл..."  +/–
Сообщение от Аноним (96), 26-Мрт-24, 14:40 
Да, есть такая проблема. И это проблема в том числе самого SQLite. Слишком многое там конфигурируется во время компиляции флагами компиляции. Потому что нефиг на Си писать. Писать надо было на плюсах и юзать шаблоны, создавая где очень критично к производительности 2 версии кода, и не через макросы, а через ifы/switchи, где в каждом варианте все ветви кроме гдной будут выоптимизированы компилятором. Ещё проблем добавляет модель разработки SQLite - "мы не будем брать ваши патчи, мы возьмём только заказ на работу и гонорар за его исполнение". Костыльный вариант решения я вижу таким — запакетировать несколько бинарных вариантов либы под распространённые use case и линковать приложения к нужному варианту.

>Но всё равно нельзя взять и подсунуть произвольную версию и только разрабы знают какая подходит

Поэтому в программы и либы приходится пихать рантайм-диагностику, анализирующую флаги сборки и фичи, где целесообразно - включающую fallbackи и генерирующую рекомендации о том, какие флаги нужны.

Ответить | Правка | Наверх | Cообщить модератору

90. "Эксперимент с использованием SQLite в качестве контейнера дл..."  +/–
Сообщение от O (?), 26-Мрт-24, 13:12 
Hey

I am sorry that you think like that. Not much tracking, rather shared with me on the Lazarus forum (IDE I sued for developing Pack), and I thought clearing some points may help some others.

I can explain more and correct your mistaken statements, but because of your way of talking, I think it won't help.

Ответить | Правка | К родителю #82 | Наверх | Cообщить модератору

94. "Эксперимент с использованием SQLite в качестве контейнера дл..."  +/–
Сообщение от Аноним (89), 26-Мрт-24, 13:30 
I can give you a simple advice that will fix all issues in your format: just admit that it was an extremily bad idea to promote it as an archive format and put noticable warnings about that everywhere: on official webpages, in git repo, etc. For the uses that don't promote it as an archive format, but as a key-value store for a local use only in a pretty trusted setup ... there are plenty of solutions, and no hype around them.
Ответить | Правка | Наверх | Cообщить модератору

83. "Эксперимент с использованием SQLite в качестве контейнера дл..."  +/–
Сообщение от Аноним (84), 26-Мрт-24, 02:33 
Some moar things I forgot to add:
* If this format gets adoption, insecure implementations in other languages will emerge. Just because it is easier and more correct to use the systemwide- or standard-lib-shipped SQLite lib rather than a custom-built one. For example a Python impl will just use the one available via import sqlite3, and it is the systemwide libsqlite3.so optimized fir speed and versatility.
* SQLite database can contain garbage, that can contain sensitive information. It can be cleaned though, but vacuuming takes a twice as large space as the file is, because for fail-recovery purposes it makes a copy first, and then clears the original file.
* SQLite databases can produce additional files alongside with database files. Journals and wrute-ahead logs. Deleting such files before they are merged into a main database file corrupts that database.
Ответить | Правка | К родителю #63 | Наверх | Cообщить модератору

92. "Эксперимент с использованием SQLite в качестве контейнера дл..."  +/–
Сообщение от O (?), 26-Мрт-24, 13:24 
Thank you for the interest.

- As long as they use the official work, it will be securely checked and verified.
- If anyone attempts to rewrite, they need to take security seriously too. Just as different implementations of JPEG have different security flaws that need to be taken care of,. And a big reason most will use the official.
- One good point about Pack is that if they use SQLite, almost all will be secure from bad memory access problems, as SQLite is much more tested and reliable.


- SQLite has secure delete too, which does not take much extra time: https://www.sqlite.org/pragma.html#pragma_secure_delete

- Your point about updating, and while doing that, those files are locked by OS. No one can delete them unless they force it.

Ответить | Правка | Наверх | Cообщить модератору

98. "Эксперимент с использованием SQLite в качестве контейнера дл..."  +/–
Сообщение от Аноним (98), 26-Мрт-24, 15:58 
>SQLite has secure delete too, which does not take much extra time: https://www.sqlite.org/pragma.html#pragma_secure_delete

The fast variant doesn't guarantee cleanup. The slow one results in additional I/O = SSD wear. And you position your archiver as a faster alternative to zip, and additional cleanup will make it less fast.

>If anyone attempts to rewrite, they need to take security seriously too

Let's say it clear — IRL security of software is not a problem of software author, it is a problem of software user. Most of software have licenses, and every sane license disclaim any liabilty. For software not having licences you get no warranty either. Software is distributed AS IS and its users are liable themselves that they have to use software written by assholes. Ones who is not OK with that have to create an own "Juche"-software, in the extreme case of total so-called "supply-chain security" - the whole stack: own research facilities, own foundries, own hardware, own microcode, own firmware, own OS, own libraries, own applications, own Internet to use their Juche-stack with (because Juche-stack will never pass Web Environment Integrity check), own everything :), and hold liability themselves and blame themselves solely. This will never happen in real world. You know it (don't pretend you don't!), because everyone, including you, uses insecure (and often - intentionally backdoored) software and hardware created by assholes.

In real worlde cannot expect that all the implementers will implement software "securely" when implementing it "securely" faces serious challenges. In this case it is very tempting to just use an SQLite lib from the distro/programming language and add a few code above it. Using another SQLite lib is not easy ... for example in the case of Python
a) one needs a dependency, `apws` package
b) since it is implemented in C, it has to be built, which is a big issue in Windows hosts
c) one needs a hardened SQLite lib
d) it also has to be built
e) one needs a way to plug that hardened SQLite lib into the programming language lib ... surprise, there is no such a feature in `apws`, its author's build scripts link it to hardcoded source-level included SQLite. There are ways to link it to external `libsqlite3.so`, but it seems they are jntentionally made pain in the ass. So one has to implement the feature in `apws` to load a certain shared SQLite lib per connection, persuade the maintainer of `apws` it is needed, upstream it to `apws`, backport that version of `apws` to all legacy versions of Python needed (and some of them are needed because assholes in PSF have decuded to drop certain versions of OSes, so to have a new version of Python one has either to maintain an own fork of it or buy a new PC with a license to a new version of the OS, and given that that OS was created by assholes, that version of the OS should never be used at all, so one has to use a hacky workaround to install a new version of Python onto a dropped version of OS that can break any time), persuade a user that he needs a tool with a dependency...

So in real world the only viable tradeoff a dev has to make is just to use `sqlite3`. And the only viable tradeoff users (including the dev himself) have to make is just use insecure shitware written by assholes in order to just not to create own software and not to withstand the pressures to create a yet another piece of shit, and pray that the files they have to open are not exploits, instead of trying to live without those files. So people will just have to use insecure impls made by assholes. Real world issues caused by the format proposed by you strongly outweight all the benefits your format claims to provide.

>One good point about Pack is that if they use SQLite, almost all will be secure from bad memory access problems, as SQLite is much more tested and reliable.

Just use Kaitai (optionally with Rust/Python/Java/C# target) for parsing of properly designed binary formats and you should be pretty safe from insecure memory access.

>Your point about updating, and while doing that, those files are locked by OS. No one can delete them unless they force it.

Mandatory locks have to be explicitly be enabled in kernel during compilation. Kernels in distros are compiled without mandatory locks. Also, I have seen a lot of times additional files being kept after normal process termination. I have to open bases with `sqlite3` CLI tool to get rid of the unneeded files in those cases (usually backup). And if a process crashes mid-operation, they are always present.

>Just as different implementations of JPEG have different security flaws that need to be taken care of,. And a big reason most will use the official.

JPEG is not based on misusing preexisting widely-available lib that is insecure for that use case. JPEG doesn't create such drastic incentives to create ihsecure shit.

If you want to create a new archive format, just leave SQLite alone and design an own format and own lib, not based on SQLite, not using its source code, but written from scratch, using memory-safe subset of languages, with security-first design, using some ideas from fast databases.

Ответить | Правка | Наверх | Cообщить модератору

Архив | Удалить

Рекомендовать для помещения в FAQ | Индекс форумов | Темы | Пред. тема | След. тема




Партнёры:
PostgresPro
Inferno Solutions
Hosting by Hoster.ru
Хостинг:

Закладки на сайте
Проследить за страницей 		Created 1996-2024 by Maxim Chirkov
Добавить, Поддержать, Вебмастеру