Memory Leak - calling itself inside of itself, forever, with a solution

Hi!
I've learned something and I've found a solution

Crawler.php

Line 202 foreach($crawling as $site)

My PHP error logs went haywire after running the script for an extended period of time, 971 occurrences of itself and it causes a crash at that point

Stack trace: #2 Crawler.php(494): Crawler->followLinks() #3 Crawler.php(494): Crawler->followLinks() #4 Crawler.php(494): Crawler->followLinks() #5 Crawler.php(494): Crawler->followLinks() . . .

and it goes on and on, up to 971 occurrences.

The issue is that , this means that what the script has been doing is
Get called for Url1 (and doesn't call getDetails for it?)
View first a href on Url1
Visit a href URL(Lets call it URL2) from Url1 and call getDetails on it
Visit the first a href on URL2. calls getdetails
visits the first a href on the last a href on url 2 from url1.
visits the first a href on the last a href on the last a href on url2 from url 1.
etc etc etc, it goes on and on forever UNTIL one subprocess doesn't have any a hrefs or all the a hrefs were processed, then it goes to its parent node.

Meaning

The first original array worth of a href URLs are not fully processed until the sub-processes finish, and for the sub-processes to finish the sub-sub-processes have to finish, and eventually you get to a point where this happens Allowed memory size of 16582912000 bytes exhausted (tried to allocate 20480 bytes)

Solution!

Line 166 function followLinks($url) to function followLinks($url, $depth = 0)
170 Insert if ($depth >= 12) {return;} replace 12 with how deep you want it to go
Line 203: erase and fill with if(isset($site)){$this->followLinks($site, $depth + 1);}

Alternative Solution

public function count_method_occurrences($method_name) { $backtrace = debug_backtrace(); $count = 0; foreach ($backtrace as $trace) { if (isset($trace['class']) && isset($trace['function']) && $trace['function'] === $method_name) { $count++; } } return $count; }

Call it via

if ($this->count_method_occurrences('followLinks') < 12) { foreach(){} }

*Please note occurrences via this will be +1 than $depth

[safesploit]

@dehlirious regarding your earlier pull request, I have some concerns, which I made comments about in the request. Just awaiting feedback, on those. Otherwise, I'm happy to merge 😄

--
@dehlirious I have made indexing changes to your post, so I can more easily trace through the code and your recommendations.

Below I have reformatted the code for GitHub readability and renamed functions and variables from snake_case to camelCase.

Alternative Solution

public function countMethodOccurrences($methodName)
{
	$backtrace = debug_backtrace();
	$count = 0;
	
	foreach ($backtrace as $trace) 
	{
		if (isset($trace['class']) && 
			isset($trace['function']) && 
			$trace['function'] === $methodName) 
		{
			$count++; 
		}
	}

	return $count;
}

Call countMethodOccurrences() via

if ($this->countMethodOccurrences('followLinks') < 12) 
{ 
	foreach()
	{
		//
	} 
}

Regarding that pull request; ignore it, I didn't realize it included several unrelated commits.

And I'm a fan of the restyling/naming. Although all I'll mention is currently I think that function could provide potential future usage but I'm just using this version myself

And I can't seem to get my code to appear formatted correctly, on my phone.

followLinks($url, $depth = 0) {
    If($depth >= 12) {
        return;
    } 
    if(isset($site)){
        $this->followLinks($site, $depth + 1);
    }
}

And on a sidenote; I have made a lot of changes and I'll be sure to try and share the useful/beneficial ones before I (attempt to) rewrite it and use Solr instead of MySQL.

[safesploit]

Check your last comment I edited, you need to use triple backticks (on separate lines) to enclose the code. See, Creating and highlighting code blocks.

Completely understand, I am in the process of rewriting an old project I wrote in snake_case to camelCase. A consistent convention is all that matters.

Sure, I appreciate you pushing updates. If you have a public branch on your forked version I can always check changes and new features there.

I presume you're using Apache Solr for full-text search, in which case I might suggest Redis. The reason I suggest Redis is besides being a multi-threaded cache and full-text search, it also has the benefit of being a database which supports replicas and edge case scenarios (such as that offered by Kubernetes).
With that opinion in mind, there are extensive evaluations for different database models and key-value pair while having amazing performance benefits initially, being denormalised can lose performance benefits when scaling. Furthermore, Redis under default configuration is not ACID compliant, a major reason for its blazing performance when compared with a relational database like MySQL.
If you're deciding between database models, I might suggest watching Which Database Model to Choose?
@HighPerformanceProgramming.
If you wish to discuss design choices further feel free to reach out, my email is on my website and LinkedIn.

Just released the changes. Check them out. Let me know what you think.
the descriptions & sentences column are going to be blank because in my instance I processed the text from the website via python not php, so I don't have a php implementation of that available. So for now the only fields that will get data for searching are "url", "title", "meta_descriptions","meta_keywords"
https://github.com/dehlirious/doogle
commit: main...dehlirious:doogle:AlteredCore

EDIT: Check it out live at https://zrr.us/doogle/search.php?term=chatgpt&type=sites
but please note that database querying time is what is making it so slow(edit: reduced from ~40s to ~8.6)
I'm open to all critique and criticisms. But do note; it's a work in progress and I haven't exactly finished it and buttoned up every aspect in the way I want to.

"I presume you're using Apache Solr for full-text search"
I haven't implemented it yet, have not looked into it well enough. Thanks for the suggestion of redis, I'll look into it after watching that video.

[safesploit]

From what I've had a chance to look at so far, I like the change. I couldn't help but notice in crawl-manual.php you've placed a couple of Udemy courses. Would I be correct to assume you want to integrate ChatGPT and implement NLP?

At scale, I can begin to see the performance limitations of using PHP and MySQL. A possible solution for the server-side language without a complete rewrite could be web assembly (WASM) as PHP is compatible. Alternatively, C++ or Go, if a rewrite is acceptable, as either compiles into machine code.

Do you intend on making your Python backend or the entire source code open-source?

For the URLs, I quite simply asked ChatGPT to generate me 50-100 URLs based on a few categories and AI/ChatGPT was one of them.

I haven't tried to implement ChatGPT, for processing the URL's data and phrases, I think it'd be way too expensive for me right now, but I was planning on having a ChatGPT response on the side of all the results.

I am currently using a NLP for processing the URLs, although at this current moment , I'm not using it to it's fullest capacity.

Also, I stopped processing the unprocessed "sentences" column and reduced load time from 40s to 4-8s, and with redis caching implemented, 200ms load time after initial search.

However, if you lookup a basic word like "a" with more than 200-300k results, still quite slow on the initial search.

Even though the MySQL query says "limit 0-20", I feel MySQL is still looking through the entirety of the database even after it's found it's 20 matches. Unsure if that is the case though but performance wise that is what it feels like.

I'm going to go through a few different options, my next test is going to be with Solr, and I'll see the performance difference.
I like how Solr can index stuff, it sounds like it should be a good bump up from my current performance.

And I could release the python file, but I don't know if it's done. Sometimes I'll write code and find out there was a more efficient way all along. Like maybe I don't need the python spaCy NLP library and I can do it in PHP or use a less advanced php NLP library and achieve the same or similar result.

With the current setup, I was able to process ~250k URLs an hour by the time I fine tuned it.
Running the python via gunicorn, multi threaded and having a bash service to run equal amounts of the crawl/crawl-manual via PHP-CLI, so 4 threads of each, and the amount per hour would increase if you had the resources to have more threads.
Prior to that I was only achieving about 2.5k/hr, so as soon as I could do 250k I had to stop because queries were so dang slow.

Currently just rewriting more base stuff, like right now I've made every submit button fetch the URL via Ajax, cleaning up my solutions, removing unnecessary code.

[safesploit]

I see, that explains it then. I completely understand the need for test data. I personally used my website as it has hyperlinks to other websites like GitHub, LinkedIn, StackOverflow and Twitter. Bleeping Computer or BBC were other good resources for indexing. But please keep in mind about your bot being treated as a DoS and banned.

I personally am hosting this on a LAN and VPS so that I won't incur additional fees. Are you using a cloud provider like AWS, Azure or GCP, where you pay for the resources you use?

I don't think I understand, can you explain 'I stopped processing the unprocessed "sentences" column'?
I'm glad to hear Redis so far is giving you a desired result.

From MySQL documentation, the LIMIT query optimisation is explained to prefer a 'full table scan' (in my opinion this wording is ambiguous for documentation). However, the documentation does seem confirm your suspicion 🤔

Remember that the full-text search feature Apache Solr offers is also offered by Redis. Mentioning Redis, regarding caching servers, if you were unaware Memcached is an option. But again since Redis 6 (which added multi-threading support) and has native replica support, there is little reason to have such a complex technology stack.

I have done that enough with feeling a solution wasn't ready. From my experience, just push to GitHub (even if not properly working) and go from there. Otherwise, the solution never feels ready for the public's eyes. My exception to this was when doing malware development, which was I very cautious about before making any of the repos public But, the community is often accepting that everything is always a 'work in progress' and tends not to criticise harshly.
From a few past projects, I'd always suggest making your web app a containerised version to make it easier for others to start using

Strictly speaking, the search engine and web crawler portions are separate. So it would be completely acceptable if you programmed each using a different language. Take for instance the modern web browser solution, client-side is using JavaScript while server-side can use Node (JavaScript), PHP, Python, etc. Hence, I would argue mixing technology stacks isn't necessarily a bad thing, just consider the complexity of maintaining and how others may perceive it (because it's open-source).

I am questioning whether Doogle's backend would benefit from a migration to another language and database though 🤔 I'm just not sure if I have the time to undergo such a project at the moment!

That Ajax solution will be a much needed improvement to usability.

But please keep in mind about your bot being treated as a DoS and banned. Did you say that knowing my website was recently labeled as "deceptive" or were you just mentioning it and this happens to be a coincidence?
Today I found out using Firefox that my domain was blacklisted. "Firefox Deceptive site ahead(zrr.us). Firefox blocked this page because it may trick you into doing something dangerous like installing software or revealing personal information like passwords or credit cards. Advisory provided by Google Safe Browsing."
Yikes, I think its because of "pocketproxy.php" , a different IP was used for crawling.
I'll try to appeal it with them but I think it's time for a new domain, unfortunate, I liked that three character domain.

I currently have a dedicated server that I use for all projects.

Sorry if what I said was lacking context, "sentences" is just the indexed URL's plaintext. I was seeing what the difference between the descriptions column and sentences column were during initial testing, when it came to results, sentences isn't a good source and resulted in a lot of false positives, so I stopped getting results from the sentences column, and it reduced load time significantly.

And I looked into the MySQL thing when I first had the suspicion and got the same answer as you. Seems it's just the way it works.

I'll think about that. Even if it was just a column or two stored in Redis/memcached, it could speed up the MySQL query for sure. You're definitely right, there's no need for a complex stack. But. I'm willing to test things out. Ultimately, I have to write the code for any implementation and as I work through that, I'll get a better understanding for what exactly I need/want/what I can do. For example,
After spending an hour or so figuring out the solarium php library and writing a rough draft of code, It is the correct course of action to optimize my usage of MySQL before switching to anything else. I already have ideas about how I could try and do it haha.

"... Otherwise, the solution never feels ready for the public's eyes"
Very true and relatable.

Personally, I'm not moving away from PHP unless there's a critical reason to. Maybe to HHVM though.
After my first trial with Solr; I'd say it's not worth the complexity when making a better system for data using MySQL could ultimately be more rewarding and it would definitely be easier for the public to use. And I don't think that the code itself is slowing anything down, the only slowdown is database queries(at least with my dataset) so switching languages wouldn't be of major benefit. I'd save the strain and focus it elsewhere.

I'll have to reach out on Twitter sometime.

[safesploit]

Afraid that it's a coincidence. Web crawlers are known to create lots of traffic from a single IP address, which in the context I was speaking about can result in some type of ban. IP address bans are the easiest, but often the most problematic due to dynamic IP addresses.
Definitely appeal, I have only just seen PocketProxy and you being the developer would know better. But the HTTP_X_FORWARDED_FOR header is revealing my true IP address when using Whoer via Pocketproxy. I discussed different proxy server implementations here as transparent, anonymous and elite proxy. However, it is behaving as a transparent proxy due to using the HTTP_X_FORWARDED_FOR header. Therefore, you could use that as grounds for your appeal.

Is your dedicated server using separate environments for each project, such as virtualisation (QEMU or VirtualBox) or containerisation (Docker/Podman)?
Also, do you have all traffic coming from the same IP address the provider gave you?

I see, completely understand that choice. When developing I was just using as much data as possible, because I was learning how web crawlers worked. But now I realise the most useful data was the domain names, titles and keywords for site searches. Naturally, image searches are another story, and being happy with that solution I never felt the need to implement shopping or video searching.

Quite unfortunately about that MySQL solution, and because of how different each SQL implementation is, using the documentation for Microsoft SQL, PostgreSQL or SQLite would not be viable!

I figured as much. Apache Solr has quite a tedious syntax to write. But I'm not necessarily saying Redis is any easier. I will need to look into that solution further myself.

Twitter is a good place for consultation. If you message me on Twitter and don't get a reply, it's more likely I haven't seen your message.

Well, pre-8pm EST, the website was still "deceptive" but I just checked and it's gone! No longer labeled as deceptive or on a phishing list. Nothing.

the HTTP_X_FORWARDED_FOR header is revealing my true IP address when using Whoer.

I'm confused about that because I went to the same url and it is just revealing the IP used to make pocketproxy requests, not my personal IP. As far as I'm aware, I don't pass HTTP_X_FORWARDED_FOR. (using Firefox and Chrome)

I use docker the most. But not always, which maybe is a habit that could be amended. I don't currently have a need for virtualization like virtualbox, on my scale. If I had to scale up, absolutely something of that nature.

If you're asking if its going through another proxy, no.

I absolutely enjoy the image search functionality, it works quite well, I haven't made a single change. Adding shopping sounds like it could be a ache overall but well worthwhile once available time arises for it.

Solr is fun/tedious. It is doable though! I've got it mostly working already. But I'm just going to figure out a way to make the MySQL version better for now, anyway.

Sounds good!

[safesploit]

I see zrr.us is now listed as safe, 5pm GMT. So congrats on the successful appeal!

HTTP_X_FORWARDED_FOR is being requested here here.

If you're that involved with Docker I would suggest looking at Podman as a more secure alternative. But I am still relatively new to the containerisation scene, so just a suggestion. I am still a bit old school and rely heavily on virtualisation, but I am gradually transitioning where appropriate.

The masonry grid layout is nice, but the performance of the framework isn't great on smartphones. Otherwise, it was a good design choice, as it saved a lot of development time.
Not that difficult actually think about the requirements or rather what Google Shopping's search parameters are.
shopping table: | Item Name | Brand | Price | Delivery | Item rating | Condition | URL |
Just as an example, might be able to drop a few unnecessary fields too. After that, it's writing the search functionality.

When I have some time I will send you some relational database suggestions for improving query time. But, Amazon Aurora is a good place to look at architecture choices as it claims to be 7x faster than MySQL server.

HTTP_X_FORWARDED_FOR is being requested here here.

Initially, I thought so too! However, upon closer inspection, the getUserIp() function is utilized solely within the logcbl() function, which is a logging method employed exclusively in this snippet on line 510~:

if (!$proxy->isValidURL($url)) {//invalid URL
	$proxy->logcbl($url);
	die("Error: The requested URL was disallowed by the server administrator.");
}

I'm open to giving Podman a try, it isn't the first time I've heard of it, and I believe virtualization is the right approach so keep to it when possible, I'd suggest.

Good to know about the framework. And you're right, it should be relatively simple to implement.

And I'll probably try multiple options, so I'm happy to hear more suggestions.

As it turns out, using a prefix wildcard in a MySQL search query disregards all indexes (including full-text ones) and results in a full table scans.
Removing the prefix "%" ($searchTerm = "%" . $parsedInput['term'] . "%";) significantly improved performance, but without extensively reorganizing the data in the descriptions column, the returned results will not be as relevant.
I think a rough comparison is 17000 results vs 130 for the same search term.
I briefly looked into it and MongoDB appears to be a potential contender in terms of performance as well.
At the moment, I'm not looking for a fully managed database.
Managed hosting would be a fantastic choice if I were running a business that required constant uptime and limitless scalability.
For now, I have time, so I want to learn how to scale it myself, maybe save a few bucks in the future.

[safesploit]

A possible table for shopping.

CREATE TABLE `shopping` (
  `id` INT(11) NOT NULL,
  `title` VARCHAR(255) NOT NULL,
  `price` INT NOT NULL,
  `url` VARCHAR(255) NOT NULL,
  `description` TEXT,
  `category` VARCHAR(255),
  `dateAdded` DATETIME NOT NULL,

-- Based on the images table
  `imageUrl` varchar(512) NOT NULL,
  `alt` varchar(512) NOT NULL,
  `title` varchar(512) NOT NULL,
  `clicks` int(11) NOT NULL DEFAULT '0',
  `broken` tinyint(4) NOT NULL DEFAULT '0',
  PRIMARY KEY (id)
);

The masonry grid layout would probably work well too. Additionally, a navigation panel on the left or right corner (taking up 15% - 25%) would be useful for the search parameters.

The following is generated template code, which may not comply with conventions used in Doogle's development.

<!DOCTYPE html>
<html>
<head>
	<title>Doogle Shopping Search</title>
	<style>
		body {
			margin: 0;
			padding: 0;
		}

		.navbar {
			position: fixed;
			top: 0;
			left: 0;
			height: 100%;
			width: 20%;
			background-color: #f2f2f2;
			overflow: hidden;
			display: flex;
			flex-direction: column;
			align-items: center;
			padding-top: 20px;
		}

		.navbar h1 {
			margin-top: 0;
			margin-bottom: 20px;
			font-size: 24px;
			color: #333;
			text-align: center;
		}

		.navbar input[type=text] {
			width: 80%;
			padding: 12px 20px;
			margin: 8px 0;
			box-sizing: border-box;
			border: 2px solid #ccc;
			border-radius: 4px;
			font-size: 16px;
		}

		.navbar button[type=submit] {
			width: 80%;
			background-color: #4CAF50;
			color: white;
			padding: 14px 20px;
			margin: 8px 0;
			border: none;
			border-radius: 4px;
			cursor: pointer;
			font-size: 16px;
		}
	</style>
</head>
<body>
	<div class="navbar">
		<h1>Shopping Website</h1>
			<form action="search.php" method="GET">
				<input type="text" placeholder="Search products..." name="search">
				<button type="submit">Search</button>
			</form>
	</div>
	<div style="margin-left: 20%;">
		<!-- rest of the page content -->
	</div>

	<script>
		function toggleNavbar() {
			var navbar = document.querySelector(".navbar");
			var hideButton = document.querySelector(".hide-button");
			navbar.classList.toggle("hidden");
			hideButton.classList.toggle("hidden");
		}
	</script>
</body>
</html>

The tabsContainer and mainResultsSection can be modified for have an additional $type = "shopping".

--

As I understand the following can be used to optimise database search time.

1. Indexing (B-tree index WHERE x BETWEEN a AND b, hash index WHERE x = y or full-text index)

2. Query optimisation (something we spoke about earlier)

3. Data normalisation

4. Limiting search results

5. Caching

6. Full-text search (Solr)

Regarding, indexing, query optimisation and limiting search results, I know those are areas of improvement. When developing Doogle I was not particularly aware of its importance.

Completely agree with that point of learning to scale yourself. Learning was the purpose behind this repo!

I sure wasn't aware of db importance either, this is the first time I've needed a potential millions of rows efficiently processed.
Right now I'm just working on side projects. This is my rough design, sidebar to doogle results.
First response is just what the user queried, and then you can extend the conversation on further. I can also add other features like image generation, audio to transcript and so on. Making all the code snippets auto highlighted was fun.
I also just recieved a GPT-4 API key from OpenAI, so I may include that, but it'd be limited to public use since it'll cost me.

Not sure if it'll stay this way though, I might have gpt responses come the same way regular results are formatted instead of next to it, or have it generate results, or a mixture,, I'm really not sure yet since the possibilities are quite endless.

Did you know "Doogle" has already been used as a search engine name and was sued by Google?

And I like that, I'm definitely down to implement such a feature but I am probably going to hold off until I've rectified query times,

With MySQL, B-tree indexes for text searches, and full-text indexes with a leading wildcard, will prevent the indexes from being utilized effectively, resulting in a full table scan which also will ignore limiting search results amount.
MongoDB, on the other hand, offers a solution for text search called text indexes. They allow you to perform searches with wildcards in any position, phrase matching, and negations.
Solr can also manage wildcards at any position within a search term.

I'm just going to have to dedicate some time to make the switch.

Sidenote!! Vector databases.

I'm so interested in vector databases, I forgot about their existence though.
Pinecone is a managed database that'll cost $0.0960/hour per 5 million vectors(estimated for 768 dimensional embeddings without metadata), it sounds interesting and it would have insane performance and results.
There is a Starter Free plan too, limited to one index and one project.

There are other providers too for vector databases but pinecone is the only one I have an account with currently.
The only issue is that vector databases are quite resource intensive, it's potentially overkill as well.

EDIT: Milvus for a self hosted solution. Can provide significant performance benefits if you have a large amount of unstructured data. but also is not designed for text search specifically

Learning is the motivation behind all my projects as well, right alongside the potential to actually have something big that people enjoy using.

[safesploit]

I understand, but once implemented to use a GPT4 API key you can always remove the key and allow others to purchase their own. Do something like $gpt4ApiKey = $gpt4ApiKeySecret where $gpt4KeySecret is part of a require('.secrets.php') which will be excluded from Git pushes via the .gitignore file. Also, do an if-statement where if the value is "" or NULL the GPT4 functionality is disabled (or put a toggle in the code).

Connecting a search engine with a generative solution would be quite interesting as the output will change each time.

I never looked into the name origin as 'Doogle' was originally named 'Google Clone'. From reading the article the company was operating 'for profit' while this project is free and open-source under a MIT License. However, Google may have copyright (intellectual property) around the logo design and/or name convention (not sure myself). So, changing the logo design, name and removing the pagination system's logos at the bottom of the page could be necessary for the future.

Facebook spent some time rectifying this very same issue where PHP and MySQL became limiting factors. Solutions to mitigate query time due to SQL limitations were explored as part of the engineering in 2005/06.

I have heard of vector databases, but never researched them. I'll do some research before I reply.