Natural materials are not sustainable or 'eco-friendly' by default !

Be it wood, stone, mud or cotton.

While some natural materials (like stone) does not involve an extensive process from extraction of various raw materials to packaged products, there are various other parameters that needs to be considered.

The raw materials, impact of sourcing the raw materials, quantifying the demand &supply vs availability of resources, Processing methods that it demands, Transportation, packaging, Geographical context of where it's being used, waste by-products(from manufacturing, processing &packaging) , and after-life of the product.

In conclusion, it makes sense to use locally available materials, 

Because one material in a certain scenario can be more sustainable than another. 

And it makes sense to put a cap on the extent of quarrying, mining and manufacturing. Like how sand mining is banned due to over extraction, and M-sand has replaced it in construction, similarly soil extraction, quarrying, mining also needs a regulation before natural resources are further depleted.