-
Notifications
You must be signed in to change notification settings - Fork 26
CIE-5505 - Having DataHub Edge doc in a single place #3944
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 7 commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
f6434f7
CIE-5505 - Having DataHub Edge doc in a single place
rthrippleton 4aa84a5
Information about NAS should go into the Edge doc
rthrippleton 41b3f9f
Broken link
rthrippleton 03bbd32
Merge remote-tracking branch 'origin/release/y2025' into CIE-5505
rthrippleton 2dbb46f
Typo
rthrippleton 8893ceb
'Root access' No is not strictly true now, you need root to install E…
rthrippleton 96265f2
Remove (some) comparisons where cloud and Edge are identical enough t…
rthrippleton 0954946
'in the Edge documentation' isn't helpful/is pointless here in context
rthrippleton 537edc8
Neater structure for the extra Messaging and DataHub requirements
rthrippleton 6cf61f8
Consistent wording
rthrippleton f831960
Whitespace fiddling
rthrippleton 1afc1a0
More whitespace
rthrippleton f8ddde2
Correct the location of the NFS/NAS mount
rthrippleton 3b68c57
Similar formatting for disk space
rthrippleton 9be630f
It's only a directory on disk for k3s, be more vague about other k8s …
rthrippleton 3d94eb8
Mention storage classes
rthrippleton 5fac2ed
Apply suggestions from code review
rthrippleton ff7e4a5
Broken links
rthrippleton e60cd2f
Remove this newly non-existing path from CODEOWNERS
rthrippleton File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
40 changes: 0 additions & 40 deletions
40
content/datahub/running-datahub-on-the-edge-bundle/datahub-edge-overview.md
This file was deleted.
Oops, something went wrong.
4 changes: 0 additions & 4 deletions
4
content/datahub/running-datahub-on-the-edge-bundle/index.html
This file was deleted.
Oops, something went wrong.
65 changes: 0 additions & 65 deletions
65
content/datahub/running-datahub-on-the-edge-bundle/operating-datahub-edge.md
This file was deleted.
Oops, something went wrong.
113 changes: 0 additions & 113 deletions
113
...nt/datahub/running-datahub-on-the-edge-bundle/setting-up-datahub-edge-on-k8s.md
This file was deleted.
Oops, something went wrong.
7 changes: 0 additions & 7 deletions
7
content/datahub/running-datahub-on-the-edge-bundle/working-with-datahub-edge.md
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| --- | ||
| weight: 80 | ||
| title: DataHub | ||
| layout: bundle | ||
| sector: | ||
| - edge_server | ||
| --- | ||
|
|
||
| {{< product-c8y-iot >}} DataHub on Edge offers the same functionality as a cloud installation of {{< product-c8y-iot >}} DataHub, and is an optional component of Edge. The significant difference is that processes and data are entirely local to your network, rather than in the cloud. You can define offloading pipelines, which regularly move data from the Operational Store of {{< product-c8y-iot >}} into a data lake. In the Edge setup, a NAS or local disk is used as data lake. Dremio, the internal engine of {{< product-c8y-iot >}} DataHub, can access the data lake and run analytical queries against its contents, using SQL as the query interface. | ||
|
|
||
| To learn more about DataHub in general, see [DataHub overview](/datahub/datahub-overview). As an end user, DataHub on Edge appears and behaves much the same as DataHub in a cloud installation, subject to the limitations in the comparison table later in this section. | ||
|
|
||
| ### Installing and Using DataHub | ||
|
|
||
| DataHub is an optional component of Edge, and can be enabled by updating the `spec.dataHub` field in the Edge custom resource (CR). For more details on the `spec.messagingService` field, refer to [Edge custom resource - DataHub](/edge-kubernetes/edge-custom-resource-definition/#k8-edge-datahub). For general guidance on configuring Edge, see the [Install Edge](/edge-kubernetes/installing-edge-on-k8/) and [Modify Edge](/edge-kubernetes/manage-edge/#modify-edge) sections in the Edge documentation. | ||
hame-c8y marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| The data lake will always be written to the host file-system under the path `/datahub/datalake`, whatever is mounted there. You are expected to have NFS or some other form of NAS file-system mounted at that path _on all nodes of the Kubernetes cluster that Edge is running on_. This is to ensure the resilience of your data lake contents. | ||
|
|
||
| In order to access Dremio, you must also make the domain `datahub-<domain_name>` resolvable, just as the configured domain name and `management-<domain_name>` were made resolvable in [Accessing Edge](/edge-kubernetes/installing-edge-on-k8/#accessing-edge). | ||
|
|
||
| ### Comparison between DataHub Edge and DataHub Cloud | ||
rthrippleton marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| | Area | {{< product-c8y-iot >}} DataHub Edge | {{< product-c8y-iot >}} DataHub Cloud | | ||
| | ----- | ----- | ----- | ----- | | ||
| | High Availability | Depending on any underlying virtualization technology | Depending on the cloud deployment setup | | ||
rthrippleton marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| | Vertical scalability | Yes | Yes | | ||
| | Horizontal scalability | No | Yes | | ||
| | Upgrades with no downtime | No | No | | ||
| | Installation | Offline & Online | Online | | ||
| | Dremio cluster setup | 1 master, 1 executor | Minimum 1 master, 1 executor | | ||
| | Data lakes | NAS or local disk | Azure Storage, S3, (NAS) | | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.