initial commit via gitinitshit
This commit is contained in:
+12
@@ -0,0 +1,12 @@
|
|||||||
|
# da soft link to latest hibp multi file dir
|
||||||
|
hibp_list
|
||||||
|
# dirs
|
||||||
|
logs
|
||||||
|
normalized
|
||||||
|
backup_files
|
||||||
|
lists
|
||||||
|
# temp, test, backup, and log files
|
||||||
|
*.tmp.txt
|
||||||
|
test_*
|
||||||
|
*.bak
|
||||||
|
*.log
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
# Notes
|
||||||
|
`hibp_sha1_dir` is a softlink to the latest haveibeenpwned sha1 multi-file dir `ln -s 20251019_haveibeenpwned_sha1 hibp_sha1_dir`
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
### Packages
|
||||||
|
`sudo apt update && sudo apt install conv ripgrep byobu -y`
|
||||||
|
### haveibeenpwned-downloader
|
||||||
|
#### dotnet
|
||||||
|
**Dotnet Installer Links**
|
||||||
|
* [Linux Install Main Page](https://learn.microsoft.com/en-us/dotnet/core/install/linux)
|
||||||
|
* [Debian Package Manager](https://learn.microsoft.com/en-us/dotnet/core/install/linux-debian?tabs=dotnet9)
|
||||||
|
* [Linux Install Script](https://learn.microsoft.com/en-us/dotnet/core/install/linux-scripted-manual)
|
||||||
|
todo: install script
|
||||||
|
packages
|
||||||
|
dotnet
|
||||||
|
debian-like
|
||||||
|
rhel-like
|
||||||
|
fall back: manual install script
|
||||||
|
haveibeenpwned-downloader
|
||||||
|
maybe: dat stupid fuckoff cancer fuckin normalizefag
|
||||||
|
todo: install instr
|
||||||
|
todo: bashrc fuckery
|
||||||
|
todo: archive
|
||||||
|
todo: script to delete earlier files and softlink downdoot da latest with auto date, create new softlink
|
||||||
|
|
||||||
|
## Downdootan from hibp
|
||||||
|
1. [Curl Method](https://github.com/HaveIBeenPwned/PwnedPasswordsDownloader/issues/79)
|
||||||
|
2. [PwnedPasswordsDownloader aka haveibeenpwned-downloader](https://github.com/HaveIBeenPwned/PwnedPasswordsDownloader)
|
||||||
|
### Commands
|
||||||
|
make da dirrr `mkdir "$(date +%Y%m%d)_haveibeenpwned_sha1"`
|
||||||
|
downdoot single file: `haveibeenpwned-downloader "$(date +%Y%m%d)_haveibeenpwned_sha1".tmp`
|
||||||
|
manual nuke files `rm -f $PWD/*_hibp_sha1.tmp.txt; rm -rf $PWD/*_haveibeenpwned_sha1`
|
||||||
|
|
||||||
|
### hibdpdl usage
|
||||||
|
todo: sanity checc thjessss
|
||||||
|
one file (./myfile.txt) sha1 `haveibeenpwned-downloader myfile`
|
||||||
|
one file (./myfile.txt) ntlm `haveibeenpwned-downloader -n myfile`
|
||||||
|
multiple files in dir (./mydir) sha1 `haveibeenpwned-downloader mydir -s false`
|
||||||
|
multiple filse in dir (./mydir) ntlm `haveibeenpwned-downloader -n mydir -s false`
|
||||||
|
|
||||||
|
## Spitballing / Scratch
|
||||||
|
1. downdoot one larg fiel
|
||||||
|
2. loop through five hex chars uppercase exclusive
|
||||||
|
3. rg for those
|
||||||
|
4. output eachi to da proper fiel :3
|
||||||
|
|
||||||
|
ls -q 20251019_haveibeenpwned_sha1 | head
|
||||||
|
head 20251019_haveibeenpwned_sha1/00000.txt
|
||||||
|
|
||||||
|
## normalizztion
|
||||||
|
dis silly sort FOOKAN WORTKS??
|
||||||
|
`awk '{val="0x" $1; print strtonum(val),$0 ;}' $tmp_file | sort -n -k 1 | sed 's/^[^ ]* //' > "$sorted_file"`
|
||||||
|
|
||||||
|
timing vers wit webhookie
|
||||||
|
`start=$(date +%s); awk '{val="0x" $1; print strtonum(val),$0 ;}' $tmp_file | sort -n -k 1 | sed 's/^[^ ]* //' > "$sorted_file"; end=$(date +%s); duration=$(($end - $start)); echo "$duration seconds" | tee -a time_log.txt; webhook "DILDO MACCARONI ITS FOOKAN DONE IN FOOKAN $duration SECONDS" true`
|
||||||
|
|
||||||
+34
@@ -0,0 +1,34 @@
|
|||||||
|
# Roadmap
|
||||||
|
## Sources
|
||||||
|
todo: links and torrents
|
||||||
|
todo: log all acquired leaks/files
|
||||||
|
|
||||||
|
## Decompression and Localiztion
|
||||||
|
decompressitalllol.sh
|
||||||
|
log all da info
|
||||||
|
recursively decompress to a new dir
|
||||||
|
decompress da files in dat dir (for recursive compressions)
|
||||||
|
log all da new file info
|
||||||
|
find all da good files and move them into a final output dir
|
||||||
|
|
||||||
|
## Normalization
|
||||||
|
todo:
|
||||||
|
do a best to make tooling to make this a bit less manual
|
||||||
|
convert encoding to whatever haveibeenpwned expects for sha1
|
||||||
|
remove duplicates
|
||||||
|
output
|
||||||
|
sql?
|
||||||
|
|
||||||
|
## Checking
|
||||||
|
test them against haveibeenpwned sha1 list
|
||||||
|
sanity checks
|
||||||
|
sha1 encoden each entry
|
||||||
|
check against list
|
||||||
|
output matches with the sha1 to good output file
|
||||||
|
log non-matching files to non-matching output file
|
||||||
|
ntlm list too? idk if its different data
|
||||||
|
|
||||||
|
## Cracking Haveibeenpwned remainders
|
||||||
|
aws gpu node
|
||||||
|
hashcat
|
||||||
|
todo: define mixed attacks
|
||||||
@@ -0,0 +1,208 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# safer bash behaviour
|
||||||
|
set -euo pipefail
|
||||||
|
IFS=$'\n\t'
|
||||||
|
|
||||||
|
# todo
|
||||||
|
## test da shit
|
||||||
|
## x output to output dir
|
||||||
|
## non-interactive modes on decompress
|
||||||
|
## handle name conflicts
|
||||||
|
## handle password protected archives
|
||||||
|
## handle nested archives
|
||||||
|
## handle more archive types
|
||||||
|
## log more info
|
||||||
|
### tee current status to stdout and a log file with timestamps
|
||||||
|
## get file encoding, size, type of output and log
|
||||||
|
## generalize so i can use it in any dir
|
||||||
|
## cmdline options for various modes
|
||||||
|
## archive testing where possible
|
||||||
|
## final output to another dir
|
||||||
|
### function to normalize file encoding to whatever haveibeenpwned expects (utgf-8?)
|
||||||
|
### recursive flattem da good files to a single dir
|
||||||
|
## frick i ahve to delete the originaal files after backing up and decompressing them
|
||||||
|
## because it'll break da recursive shit igg
|
||||||
|
## fuuuuck gonna have to sha1sum every fuckin file with the file list at all stages of decompression
|
||||||
|
### must document da info to prevent redundant effortt :pensiveanimated:
|
||||||
|
### actually check what will be includede with torrents see if can use dat hash algo?
|
||||||
|
## add package list
|
||||||
|
|
||||||
|
backup=./backup_file # backups of the archives
|
||||||
|
log_dir=./logs # logs
|
||||||
|
out_dir=./normalized # where the decomressed and normalizerd files go
|
||||||
|
iterations=3 # recursive decompress levels (can override via env var ITERATIONS)
|
||||||
|
ingress=/mnt/e/Breachsilly_Ingress # can override via env var INGRESS (use WSL-style path)
|
||||||
|
|
||||||
|
timestamp=$(date +%Y%m%d_%H%M%S)
|
||||||
|
error_log="$log_dir/decompressitalllol_error_$timestamp.log"
|
||||||
|
file_extensions="$log_dir/file_extensions_$timestamp.txt"
|
||||||
|
file_tree="$log_dir/file_tree_$timestamp.txt"
|
||||||
|
file_types="$log_dir/file_types_$timestamp.txt"
|
||||||
|
|
||||||
|
# setup
|
||||||
|
setup_silly () {
|
||||||
|
## maek da dirsa
|
||||||
|
mkdir -p "$backup/tar.gz"
|
||||||
|
mkdir -p "$backup/tar.bz2"
|
||||||
|
mkdir -p "$backup/tar.xz"
|
||||||
|
mkdir -p "$backup/gz"
|
||||||
|
mkdir -p "$backup/bz2"
|
||||||
|
mkdir -p "$backup/tar"
|
||||||
|
mkdir -p "$backup/rar"
|
||||||
|
mkdir -p "$backup/zip"
|
||||||
|
mkdir -p "$backup/7z"
|
||||||
|
mkdir -p "$backup/xz"
|
||||||
|
mkdir -p "$log_dir"
|
||||||
|
mkdir -p "$out_dir"
|
||||||
|
mkdir -p "$ingress"
|
||||||
|
}
|
||||||
|
|
||||||
|
# setup silly/get info
|
||||||
|
info_silly () {
|
||||||
|
## get a list of da file extensions
|
||||||
|
rm -f ext.tmp || true
|
||||||
|
while IFS= read -r -d '' f; do
|
||||||
|
# use basename to avoid full path issues
|
||||||
|
b="$(basename "$f")"
|
||||||
|
# only print if there's a dot
|
||||||
|
if [[ "$b" == *.* ]]; then
|
||||||
|
echo "${b##*.}" >> ext.tmp
|
||||||
|
fi
|
||||||
|
done < <(find "$ingress" -type f -print0 2>>"$error_log")
|
||||||
|
sort -u ext.tmp > "$file_extensions" 2>>"$error_log" || true
|
||||||
|
rm -f ext.tmp 2>>"$error_log" || true
|
||||||
|
|
||||||
|
## get a list of da filetypes and encodings
|
||||||
|
: > "$file_types"
|
||||||
|
while IFS= read -r -d '' f; do
|
||||||
|
file --mime-type --mime-encoding "$f" >> "$file_types" 2>>"$error_log" || true
|
||||||
|
done < <(find "$ingress" -type f -print0 2>>"$error_log")
|
||||||
|
|
||||||
|
## get a simple file tree
|
||||||
|
tree "$ingress" > "$file_tree" 2>>"$error_log" || true
|
||||||
|
}
|
||||||
|
|
||||||
|
# backup all da extant archive doots
|
||||||
|
backup_silly () {
|
||||||
|
# get da tree and file info and suc h
|
||||||
|
info_silly
|
||||||
|
## tar.gz
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar.gz" -print -exec cp -v {} "$backup/tar.gz/" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
## tar.bz2
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar.bz2" -print -exec cp -v {} "$backup/tar.bz2/" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
## tar.xz
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar.xz" -print -exec cp -v {} "$backup/tar.xz/" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
## gz
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.gz" -print -exec cp -v {} "$backup/gz/" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
## bz2
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.bz2" -print -exec cp -v {} "$backup/bz2/" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
## tar
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar" -print -exec cp -v {} "$backup/tar/" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
## rar
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.rar" -print -exec cp -v {} "$backup/rar/" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
## zip
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.zip" -print -exec cp -v {} "$backup/zip/" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
## 7z
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.7z" -print -exec cp -v {} "$backup/7z/" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
## xz
|
||||||
|
find $ingress \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -iname "*.xz" -print -exec cp {} "$backup/xz" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
}
|
||||||
|
|
||||||
|
# recursively decompress all da things to pwd
|
||||||
|
decompress_silly () {
|
||||||
|
## do all the tar.x ones first to prevent issues
|
||||||
|
### tar.gz
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar.gz" -print -exec tar -xvzf "{}" -C "$out_dir" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
### tar.bz2
|
||||||
|
find $ingress \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -iname "*.tar.bz2" -print -exec tar -xvjf "{}" -C "$out_dir" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
### tar.xz
|
||||||
|
find $ingress \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -iname "*.tar.xz" -print -exec tar -xvJf "{}" -C "$out_dir" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
## now do the rest
|
||||||
|
### gz
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.gz" -print -exec bash -c 'f="$1"; out="$2"; gzip -dc "$f" > "${out}/$(basename "${f%.gz}")"' _ {} "$out_dir" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
### bz2
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.bz2" -print -exec bash -c 'f="$1"; out="$2"; bunzip2 -c "$f" > "${out}/$(basename "${f%.bz2}")"' _ {} "$out_dir" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
### tar
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar" -print -exec tar -xvf "{}" -C "$out_dir" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
### rar
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.rar" -print -exec unrar x -o+ "{}" "$out_dir" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
### zip
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.zip" -print -exec unzip -o -d "$out_dir" "{}" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
### 7z
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.7z" -print -exec 7z x "{}" -o"$out_dir" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
### xz
|
||||||
|
### test dis
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.xz" -print -exec bash -c 'f="$1"; out="$2"; unxz -c "$f" > "${out}/$(basename "$f")"' _ {} "$out_dir" 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
}
|
||||||
|
|
||||||
|
# mvoe extracted useful loookingh files to $out_dir
|
||||||
|
move_silly () {
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -type f \( -name "*.txt" -o -name "*sql" -o -name "*csv" \) -print0 | while IFS= read -r -d '' f; do
|
||||||
|
mv -v "$f" "$out_dir/" 2>>"$error_log" || true
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# delete da originaal archives
|
||||||
|
delete_silly () {
|
||||||
|
## tar.gz
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar.gz" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
## tar.bz2
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar.bz2" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
## tar.xz
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar.xz" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
## gz
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.gz" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
## bz2
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.bz2" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
## tar
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.tar" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
## rar
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.rar" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
## zip
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.zip" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
## 7z
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.7z" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
## xz
|
||||||
|
find "$ingress" \( -path "$backup" -o -path "$log_dir" -o -path "$out_dir" -o -path "./.git" \) -prune -o -iname "*.xz" -print -exec rm -v {} 2>>"$error_log" \; 2>>"$error_log"
|
||||||
|
}
|
||||||
|
|
||||||
|
compress_silly () {
|
||||||
|
# compress the normalized files with 7z
|
||||||
|
# except RockYou2024.txt and Princess_Pi*txt which are compressed with gzip
|
||||||
|
# deletes the uncompressed output files
|
||||||
|
## these are done with gzip because that's the compression that hashcat supports
|
||||||
|
# gzip specific known files if they exist
|
||||||
|
for f in "PrincessPiV8.txt" "PrincessPi_Rules_Dictionary_2.5.txt" "RockYou2024.txt"; do
|
||||||
|
if [[ -f "$out_dir/$f" ]]; then
|
||||||
|
gzip -9 -v "$out_dir/$f" 2>>"$error_log" || true
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# Pack remaining files (excluding already-gzipped) into a single 7z archive
|
||||||
|
target_archive="$out_dir/normalized.7z"
|
||||||
|
rm -f "$target_archive" || true
|
||||||
|
find "$out_dir" -type f ! -name '*.gz' -print0 | xargs -0 -I{} 7z a -t7z "$target_archive" "{}" 2>>"$error_log" || true
|
||||||
|
# clean up uncompressed fiies
|
||||||
|
# delete_silly
|
||||||
|
}
|
||||||
|
|
||||||
|
# run da functinoszzz
|
||||||
|
setup_silly
|
||||||
|
|
||||||
|
# backup_silly
|
||||||
|
|
||||||
|
# decompression option
|
||||||
|
for((i=0; i<$iterations; i++)) {
|
||||||
|
info_silly
|
||||||
|
decompress_silly
|
||||||
|
}
|
||||||
|
|
||||||
|
# move wanted files to output dir
|
||||||
|
move_silly
|
||||||
|
|
||||||
|
# decompression option
|
||||||
|
# delete_silly
|
||||||
|
|
||||||
|
# compression option
|
||||||
|
# compress_silly
|
||||||
@@ -0,0 +1,145 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
set -e # dont waste my asstime if errros bitch
|
||||||
|
# da timestiampppy liek 20251020
|
||||||
|
timestamp=$(date +%Y%m%d)
|
||||||
|
# fiels
|
||||||
|
## this is both the downloaded fiel and da file to sort
|
||||||
|
## da fuckin genetal wards dotnet app auto appends .txt to the file name jesus fuck
|
||||||
|
download_file="lists/${timestamp}_sorted_download_single_sha1" # again no txt because the dotnet app is a form of sexually transmitted infectoion
|
||||||
|
# count total lines for progress tracking
|
||||||
|
echo "countin total lines..." | tee -a "$info_log"
|
||||||
|
total_lines=$(wc -l < "$download_file.txt")
|
||||||
|
echo "total lines to process: $total_lines" | tee -a "$info_log"
|
||||||
|
|
||||||
|
# for progress updates every 1%
|
||||||
|
update_interval=$((total_lines / 100))
|
||||||
|
[[ $update_interval -lt 1 ]] && update_interval=1
|
||||||
|
last_update=0
|
||||||
|
processed_lines=0
|
||||||
|
softlink_name=hibp_list
|
||||||
|
# backup_file=bak/test_sorted_single_sha1.tmp.txt.7z # donut need dis oc unless bein stoopd wit da backup ig
|
||||||
|
# lougs
|
||||||
|
info_log=info_log.log
|
||||||
|
error_log=error_log.log
|
||||||
|
|
||||||
|
# ig measure how long each fluttercum takes for fun idk
|
||||||
|
time_it () {
|
||||||
|
echo "startin timer"
|
||||||
|
start=$(date +%s)
|
||||||
|
echo "runnan $1 at $(date)"
|
||||||
|
eval "$1" 2>>"$error_log" # also make sure to get dem errorz if dey cum
|
||||||
|
echo "done runnan $1 at $(date)"
|
||||||
|
end=$(date +%s)
|
||||||
|
duration=$(($end - $start))
|
||||||
|
echo "$(date): finished $1 in $duration seconds" | tee -a "$info_log"
|
||||||
|
}
|
||||||
|
|
||||||
|
break_apart () {
|
||||||
|
# break the sortd file in ./lists into individual txt files named for the first five characters of the hashes
|
||||||
|
echo "breakin apart da sorted fiel into smol pieces" | tee -a "$info_log"
|
||||||
|
|
||||||
|
# make sure hibp_sha1_dir exists and is empty
|
||||||
|
mkdir -p hibp_sha1_dir
|
||||||
|
rm -f hibp_sha1_dir/*.txt 2>>"$error_log"
|
||||||
|
((processed_lines++))
|
||||||
|
|
||||||
|
# show progress every 1% or 100k lines
|
||||||
|
if ((processed_lines - last_update >= update_interval)); then
|
||||||
|
percent=$((processed_lines * 100 / total_lines))
|
||||||
|
echo "processed $processed_lines / $total_lines lines ($percent%)" | tee -a "$info_log"
|
||||||
|
last_update=$processed_lines
|
||||||
|
fi
|
||||||
|
|
||||||
|
# read da big file line by line and split by first 5 chars
|
||||||
|
# use a buffer to reduce disk writes - only write when prefix changes or buffer gets big
|
||||||
|
current_prefix=""
|
||||||
|
buffer=""
|
||||||
|
buffer_size=0
|
||||||
|
max_buffer=10000
|
||||||
|
|
||||||
|
while IFS= read -r line; do
|
||||||
|
# get first 5 chars of the hash
|
||||||
|
echo "finished! processed $total_lines lines into $file_count files in hibp_sha1_dir" | tee -a "$info_log"
|
||||||
|
|
||||||
|
# if prefix changed or buffer full, write the current buffer
|
||||||
|
if [[ "$prefix" != "$current_prefix" ]] || ((buffer_size >= max_buffer)); then
|
||||||
|
if [[ -n "$current_prefix" && -n "$buffer" ]]; then
|
||||||
|
echo -n "$buffer" > "hibp_sha1_dir/${current_prefix}.txt"
|
||||||
|
buffer=""
|
||||||
|
buffer_size=0
|
||||||
|
fi
|
||||||
|
current_prefix="$prefix"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# add to buffer
|
||||||
|
buffer+="${line}\n"
|
||||||
|
((buffer_size++))
|
||||||
|
|
||||||
|
done < "$download_file.txt"
|
||||||
|
|
||||||
|
# write final buffer if any
|
||||||
|
if [[ -n "$current_prefix" && -n "$buffer" ]]; then
|
||||||
|
echo -n "$buffer" > "hibp_sha1_dir/${current_prefix}.txt"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# count how many files we made
|
||||||
|
file_count=$(find hibp_sha1_dir -type f -name "*.txt" | wc -l)
|
||||||
|
echo "made $file_count files in hibp_sha1_dir" | tee -a "$info_log"
|
||||||
|
}
|
||||||
|
|
||||||
|
compress_run_silly () {
|
||||||
|
# compress the files in ./lists to one 7z archive
|
||||||
|
7z a ./hibp_sha1_individual_sorted_$timestamp.7z hibp_sha1_dir
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "startan scripty at $(date)"
|
||||||
|
|
||||||
|
# check for remaining bitchtxt and kill them if found >:34
|
||||||
|
if [ -f lists/*.txt ]; then
|
||||||
|
echo "i fonud sum txts in the lists dir, im nuken em lmao, list beloow" | tee -a $info_log
|
||||||
|
ls -la lists/*.txt | tee -a $info_log
|
||||||
|
rm -f lists/*.txt 2>> $error_log
|
||||||
|
fi
|
||||||
|
|
||||||
|
# do da same for dat silly soft linkl :pope:
|
||||||
|
if [ -f $softlink_name ]; then
|
||||||
|
echo "fonud softline $softlink_name nukin it" | tee -a $info_log
|
||||||
|
rm -f $softlink_name 2>> $error_log
|
||||||
|
fi
|
||||||
|
|
||||||
|
# download da silly thing
|
||||||
|
echo "startan downdoot"
|
||||||
|
time_it "haveibeenpwned-downloader $download_file"
|
||||||
|
|
||||||
|
# sort it good jus to maek sure
|
||||||
|
## dis silly widdle scrript sorts da file in place :3
|
||||||
|
## it ver silly :3
|
||||||
|
echo "startin da sortfuck"
|
||||||
|
time_it "awk '{val=\"0x\" \$1; print strtonum(val),\$0 ;}' $download_file.txt | sort -n -k 1 | sed 's/^[^ ]* //'"
|
||||||
|
|
||||||
|
# break da file into smol pieces by hash prefix
|
||||||
|
time_it "break_apart"
|
||||||
|
|
||||||
|
# zip em up heuhuehue
|
||||||
|
compress_run_silly
|
||||||
|
|
||||||
|
# nao we maek dat convenient softlinkie
|
||||||
|
# s softlink, f force overwrite existing softlink, n no dereference treat as dir
|
||||||
|
# ls -sfn $download_file $softlink_name 2>> $error_log # for dir
|
||||||
|
ls -sf $download_file $softlink_name 2>> $error_log # for file
|
||||||
|
|
||||||
|
# donut do dis when little space it get mad as fuck
|
||||||
|
# 7z backup
|
||||||
|
# echo "bakkan it up ig"
|
||||||
|
# time_it "7z a $backup_file $download_file.txt"
|
||||||
|
|
||||||
|
# send silly webhookfuckin
|
||||||
|
echo "sendin webhookfuckin"
|
||||||
|
webhook "DILDO MACCARONI ITS FOOKAN DONE at $(date)" true
|
||||||
|
|
||||||
|
# mayhaps reboot
|
||||||
|
## 1 minute delay to make sure shit fucks
|
||||||
|
echo "doin an rebootasn in an 1 mins or sumtin"
|
||||||
|
sudo shutdown -r +1 "REBOOTFUCKIN IN 10 MINS BITCH"
|
||||||
|
|
||||||
|
echo ":awooo: done :3"
|
||||||
@@ -0,0 +1,74 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# todo:
|
||||||
|
## auto byobu?
|
||||||
|
|
||||||
|
# to use dis i need to do checks for the dfiles that cleanup_environ cleans
|
||||||
|
# set -e # stahp on any errpr
|
||||||
|
|
||||||
|
# all thos silly hex sillyvalues from 0x00000 to 0xFFFFF
|
||||||
|
# to be used to split the hibp sha1 file into multiple files
|
||||||
|
# each file containing all sha1 hashes starting with that sillyvalue
|
||||||
|
dec_max=1048575
|
||||||
|
start_val_hex=0x00000
|
||||||
|
end_val_hex=0xFFFFF
|
||||||
|
increment_hex=0x00001
|
||||||
|
|
||||||
|
# da file shit
|
||||||
|
timestamp=$(date +%Y%m%d)
|
||||||
|
# tmp file values
|
||||||
|
## this is the value fed to haveibeenpwned-downloader
|
||||||
|
## it automatically appends .txt to da filename
|
||||||
|
tmp_file="$PWD/${timestamp}_hibp_sha1.tmp"
|
||||||
|
## this is the txt file created by haveibeenpwned-downloader
|
||||||
|
tmp_file_txt="$tmp_file.txt"
|
||||||
|
new_dir="$PWD/${timestamp}_haveibeenpwned_sha1"
|
||||||
|
softlink_name="$PWD/hibp_sha1_dir"
|
||||||
|
softlink_target="$new_dir"
|
||||||
|
|
||||||
|
setup_environ () {
|
||||||
|
# cleanup environment
|
||||||
|
# fail quietly if no such files/dirs exist
|
||||||
|
## nuek any temp files
|
||||||
|
# rm -f $PWD/*_hibp_sha1.tmp.txt 2>/dev/null
|
||||||
|
## delet any old hibp sha1 dirs
|
||||||
|
# rm -rf $PWD/*_haveibeenpwned_sha1 2>/dev/null
|
||||||
|
|
||||||
|
# make new dir for new hibp sha1 files
|
||||||
|
mkdir -p "$new_dir" # exit quietly if exists
|
||||||
|
}
|
||||||
|
|
||||||
|
finish_cleanup () {
|
||||||
|
# nuke da temp file
|
||||||
|
rm -f "$tmp_file_txt"
|
||||||
|
# reboot in 2 mins to clear any cached
|
||||||
|
sudo shutdown -r +2
|
||||||
|
}
|
||||||
|
|
||||||
|
downdoot_file () {
|
||||||
|
# downdoot da fuckin file
|
||||||
|
haveibeenpwned-downloader "$tmp_file"
|
||||||
|
}
|
||||||
|
|
||||||
|
# fuckin do the hell
|
||||||
|
do_the_fucking_split () {
|
||||||
|
# loop throught a bajuilli9opn fuckin numbs for hex bitch
|
||||||
|
for((i=0; i<=$dec_max; i++)) {
|
||||||
|
current_hex=$(printf "0x%05X" $i)
|
||||||
|
echo "$current_hex"
|
||||||
|
|
||||||
|
# do da fuckin hunt and shit fuckbitchfuck
|
||||||
|
rg --no-line-number "^$current_hex" "$tmp_file_txt" > "$new_dir/$current_hex.txt"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
post_process_files () {
|
||||||
|
# softlink
|
||||||
|
# s softlink, f force overwrite existing softlink, n no dereference treat as dir
|
||||||
|
ln -sfn "$softlink_target" "$softlink_name"
|
||||||
|
}
|
||||||
|
|
||||||
|
setup_environ
|
||||||
|
downdoot_file
|
||||||
|
do_the_fucking_split
|
||||||
|
post_process_files
|
||||||
|
# finish_cleanup
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
smbencrypt dingle | awk '{print $1}' | tail -1
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
echo "$1" | sha1sum | awk '{print $1}' | tr 'a-z' 'A-Z'
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# fiels
|
||||||
|
input_file=./test_sha1.tmp.txt
|
||||||
|
sorted_file=./test_SORTED_sha1.tmp.txt
|
||||||
|
backup_file=./bak/20251020_single_sha1.txt.7z
|
||||||
|
# lougs
|
||||||
|
info_log=./info_log.log
|
||||||
|
error_log=./error_log.log
|
||||||
|
|
||||||
|
# start da timer
|
||||||
|
echo "startin timer"
|
||||||
|
start=$(date +%s)
|
||||||
|
|
||||||
|
# do da magical sortfuck
|
||||||
|
echo "performin da magical sortfuck"
|
||||||
|
# awk '{val="0x" $1; print strtonum(val),$0 ;}' "$input_file" | sort -n -k 1 | sed 's/^[^ ]* //' > "$sorted_file" 2>> "$error_log"
|
||||||
|
awk '{val="0x" $1; print strtonum(val),$0 ;}' $input_file | sort -n -k 1 | sed 's/^[^ ]* //' > $sorted_file
|
||||||
|
echo "finished da magical sortfuck! retcode $?"
|
||||||
|
|
||||||
|
# end da timer and caluclate duratrion
|
||||||
|
echo "endin timer"
|
||||||
|
end=$(date +%s)
|
||||||
|
duration=$(($end - $start))
|
||||||
|
echo "sorted (in theoty) in $duration seconds"
|
||||||
|
|
||||||
|
# updoot da log
|
||||||
|
## can reuse dis in a function or sumshit
|
||||||
|
echo "loggin shit"
|
||||||
|
echo "$(date): $input_file sorted to $sorted_file in $duration seconds" | tee -a "$info_log"
|
||||||
|
|
||||||
|
# send silly webhookfuckin
|
||||||
|
echo "sendin webhookfuckin"
|
||||||
|
webhook "DILDO MACCARONI ITS FOOKAN DONE IN FOOKAN $duration SECONDS" true
|
||||||
|
|
||||||
|
# maek an backup
|
||||||
|
echo "maekan a backup"
|
||||||
|
7z a "$backup_file" "$input_file"
|
||||||
|
|
||||||
|
# mayhaps reboot
|
||||||
|
## 10 minute delay to make sure shit fucks
|
||||||
|
echo "doin an rebootasn in 10 mins or sumtin"
|
||||||
|
sudo shutdown -r +10 "REBOOTFUCKIN IN 10 MINS BITCH"
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# fiels
|
||||||
|
input_file=./test_sha1.tmp.txt
|
||||||
|
sorted_file=./test_SORTED_sha1.tmp.txt
|
||||||
|
# lougs
|
||||||
|
info_log=./info_log.log
|
||||||
|
error_log=./error_log.log
|
||||||
|
|
||||||
|
# maek an backup
|
||||||
|
echo "maekan a backup"
|
||||||
|
cp "$input_file" "$input_file.bak"
|
||||||
|
|
||||||
|
# start da timer
|
||||||
|
echo "startin timer"
|
||||||
|
start=$(date +%s)
|
||||||
|
|
||||||
|
# do da magical sortfuck
|
||||||
|
echo "performin da magical sortfuck"
|
||||||
|
awk '{val="0x" $1; print strtonum(val),$0 ;}' "$input_file" | sort -n -k 1 | sed 's/^[^ ]* //' > "$sorted_file" 2>> "$error_log"
|
||||||
|
echo "finished da magical sortfuck! retcode $?"
|
||||||
|
|
||||||
|
# end da timer and caluclate duratrion
|
||||||
|
echo "endin timer"
|
||||||
|
end=$(date +%s)
|
||||||
|
duration=$(($end - $start))
|
||||||
|
echo "sorted (in theoty) in $duration seconds"
|
||||||
|
|
||||||
|
# updoot da log
|
||||||
|
## can reuse dis in a function or sumshit
|
||||||
|
echo "loggin shit"
|
||||||
|
echo "$(date): $input_file sorted to $sorted_file in $duration seconds" | tee -a "$info_log"
|
||||||
|
|
||||||
|
# send silly webhookfuckin
|
||||||
|
echo "sendin webhookfuckin"
|
||||||
|
webhook "DILDO MACCARONI ITS FOOKAN DONE IN FOOKAN $duration SECONDS" true
|
||||||
|
|
||||||
|
# mayhaps reboot
|
||||||
|
## 10 minute delay to make sure shit fucks
|
||||||
|
echo "doin an rebootasn in 10 mins or sumtin"
|
||||||
|
sudo shutdown -r +10 "REBOOTFUCKIN IN 10 MINS BITCH"
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
tar gzip bzip2 rar zip 7z xz conv ripgrep screen byobu
|
||||||
|
dotnet
|
||||||
|
haveibeenpwned-downloader
|
||||||
@@ -0,0 +1,74 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
IFS=$'\n\t'
|
||||||
|
|
||||||
|
# Default settings
|
||||||
|
output_encoding="utf-8"
|
||||||
|
hibp_sha1_dir="${HIBP_SHA1_DIR:-hibp_sha1_dir}" # Can override with env var
|
||||||
|
|
||||||
|
# Check for required tools
|
||||||
|
for cmd in file iconv sha1sum rg; do
|
||||||
|
if ! command -v "$cmd" >/dev/null 2>&1; then
|
||||||
|
echo "Error: Required command '$cmd' not found" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# Validate arguments
|
||||||
|
if [[ $# -eq 0 ]]; then
|
||||||
|
echo "Usage: $(basename "$0") <password> [debug]" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
input_str="$1"
|
||||||
|
debug_mode=0
|
||||||
|
[[ ${2:-} == "debug" ]] && debug_mode=1
|
||||||
|
|
||||||
|
# Validate HIBP directory exists
|
||||||
|
if [[ ! -d "$hibp_sha1_dir" ]]; then
|
||||||
|
echo "Error: HIBP SHA1 directory '$hibp_sha1_dir' not found" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
# detect da input encoding
|
||||||
|
input_encoding="$(echo -n "$input_str" | file -i - | awk -F'charset=' '{print $2}' || echo 'utf-8')"
|
||||||
|
[[ -z "$input_encoding" ]] && input_encoding="utf-8"
|
||||||
|
|
||||||
|
# convert da silly into utf-8
|
||||||
|
converted_str=$(iconv -f "$input_encoding" -t "$output_encoding" <<< "$input_str" 2>/dev/null || echo "$input_str")
|
||||||
|
|
||||||
|
# convert da silly into sha1
|
||||||
|
sha1_str=$(echo "$converted_str" | sha1sum | awk '{print $1}' | tr 'a-z' 'A-Z')
|
||||||
|
|
||||||
|
# get da first five chars
|
||||||
|
first_five="${sha1_str:0:5}"
|
||||||
|
|
||||||
|
# lookup da hash in hibp sha1 dir
|
||||||
|
lookup_file="$hibp_sha1_dir/$first_five.txt"
|
||||||
|
|
||||||
|
if ((debug_mode)); then
|
||||||
|
echo "input_str (\$1): $input_str"
|
||||||
|
echo "hibp_sha1_dir: $hibp_sha1_dir"
|
||||||
|
echo "output_encoding: $output_encoding"
|
||||||
|
echo "input_encoding: $input_encoding"
|
||||||
|
echo "converted_str: $converted_str"
|
||||||
|
echo "sha1_str: $sha1_str"
|
||||||
|
echo "first_five: $first_five"
|
||||||
|
echo "lookup_file: $lookup_file"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check if lookup file exists
|
||||||
|
if [[ ! -f "$lookup_file" ]]; then
|
||||||
|
echo "Hash prefix $first_five not found in database" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Look up the hash (ripgrep returns 0 if found, 1 if not found)
|
||||||
|
if rg --quiet --fixed-strings "$sha1_str" "$lookup_file" 2>/dev/null; then
|
||||||
|
echo "Password found in breaches"
|
||||||
|
exit 0
|
||||||
|
else
|
||||||
|
echo "Password not found in breaches"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
awk '{val="0x" $1; print strtonum(val),$0 ;}' your_file.txt | sort -n -k 1 | sed 's/^[^ ]* //'
|
||||||
|
|
||||||
|
```
|
||||||
|
#!/bin/bash
|
||||||
|
# fiels
|
||||||
|
input_file="$PWD/test_sha1.tmp.txt"
|
||||||
|
sorted_file="$PWD/test_SORTED_sha1.tmp.txt"
|
||||||
|
log_file="$PWD/info.log"
|
||||||
|
|
||||||
|
# start da timer
|
||||||
|
start=$(date +%s)
|
||||||
|
|
||||||
|
# do da magical sortfuck
|
||||||
|
awk '{val="0x" $1; print strtonum(val),$0 ;}' "$input_file" | sort -n -k 1 | sed 's/^[^ ]* //' > "$sorted_file"
|
||||||
|
end=$(date +%s)
|
||||||
|
|
||||||
|
# end da timer and caluclate duratrion
|
||||||
|
duration=$(($end - $start))
|
||||||
|
|
||||||
|
# updoot da log
|
||||||
|
echo "$(date): $input_file sorted to $sorted_file in $duration seconds" | tee -a "$log_file"
|
||||||
|
|
||||||
|
# send silly webhookfuckin
|
||||||
|
webhook "DILDO MACCARONI ITS FOOKAN DONE IN FOOKAN $duration SECONDS" true
|
||||||
|
```
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
auto start in byobu
|
||||||
|
environment check
|
||||||
|
packages
|
||||||
|
dotnet
|
||||||
|
downdooter
|
||||||
|
files in current dir
|
||||||
|
interactively ask to delete files
|
||||||
|
speed up rg
|
||||||
|
test time
|
||||||
|
nosleep
|
||||||
|
sanity checks
|
||||||
|
hash lengths
|
||||||
|
successful sorting in txt files
|
||||||
|
sum kinda bettar progress indicator
|
||||||
|
timing for estimated completion
|
||||||
|
|
||||||
|
sort then split! # literally fucking must do this to have any god damned chance
|
||||||
|
# DO SOME MORE RESEARCH
|
||||||
|
# also prepends da 0x remov
|
||||||
|
awk '{val="0x" $1; print strtonum(val),$0 ;}' your_file | sort -n -k 1 | sed 's/^[^ ]* //'
|
||||||
|
|
||||||
|
# another option
|
||||||
|
## does this assume 0x??
|
||||||
|
paste -d: <(echo 'ibase=16'; cut -d' ' -f1 your_file | bc) your_file | sort -t: -k1,1n | cut -d: -f2-
|
||||||
|
|
||||||
|
# opttion?
|
||||||
|
## prepend 0x to each line?
|
||||||
|
## -g is GENRAL numerical sort and it can handle hex nativelu
|
||||||
|
## does not work will have to prepend 0x
|
||||||
|
sort -o outfilename -g filename
|
||||||
Reference in New Issue
Block a user