Deployment State
The deployment state records what the last successful deployment to an
environment sent: the commit of each item type, and the hash and folder of
each item. With it, deploy_all_items() deploys only what changed since
then, and reconcile_items() tells drift in the workspace from a
deployment still to run. It is written only when every item of a run
succeeded, and it holds no secret.
A state backend keeps the states, one per environment. Pass it to
deploy_all_items(), plan_all_items() or reconcile_items() with the
environment's name:
report = deploy_all_items(
"Sales-PRD",
staging,
start_path=staging,
repository_path="workspace",
state_backend=OneLakeStateBackend("Ops", "DeploymentState"),
environment="PRD",
)
In a OneLake lakehouse
OneLakeStateBackend(workspace, lakehouse) keeps each state as
Files/pyfabricops/state/<environment>.json in a lakehouse, and survives
every CI run. Keep the lakehouse in a workspace of its own, such as one for
operations: in the workspace deployed to, reconcile_items() would report
it as unmanaged.
- The identity that deploys needs to write to that lakehouse, as a
Contributor of its workspace. Its token for OneLake comes from
set_auth_provider(), as for the Fabric API. - The workspace and the lakehouse are given by name or ID, looked up once, then reached by ID.
folderchanges the folder underFiles.endpointtakes a regional endpoint, such ashttps://westus-onelake.blob.fabric.microsoft.com, so the state stays in the region of the lakehouse's capacity.- A state is saved only over the one the run read. When another run saved one in between, the save fails and the other run's state is kept:
Deployment state 'PRD' changed in Ops/DeploymentState/Files/pyfabricops/state/PRD.json since this run read it, so it was not overwritten: another run deployed to the environment meanwhile.
In a local folder
LocalJsonStateBackend(folder) keeps each state as
<folder>/<environment>.json. Keep the folder between runs: without it, a
run deploys every item, which is safe but slower. A CI cache is not enough
for that, as GitHub drops a cache unused for 7 days, and Azure DevOps keeps
one per pipeline.
Locks
A deployment holds the lock of its environment from before it reads the
state until after it records it, so two runs never deploy to one
environment at a time. Both backends keep the lock next to the state, as
<environment>.lock, which says who holds it and until when:
DeploymentLockedError: Deployment state 'PRD' is locked, held by runner@ci-host (GitHub Actions run 1234) since 2026-09-25T10:00:00Z, until 2026-09-25T12:00:00Z. Wait for that run to finish; if it is gone, call force_unlock('PRD') on the state backend.
- A run fails at once when another holds the lock, before it reads or
deploys anything.
lock_timeout=<seconds>on the backend waits for the lock instead. - A lock holds for
lock_ttlseconds, two hours by default, unless its run releases it; a failed run releases it too. After that another run takes it over, so a run that died holding it blocks the environment for no longer. force_unlock(environment)removes the lock whoever holds it, for a run that is gone. Make sure no run is deploying first.
OneLakeStateBackend("Ops", "DeploymentState").force_unlock("PRD")
plan_all_items()andreconcile_items()only read the state, and take no lock.- A lock holds no secret: the user and host that took it, the CI run when there is one, and when it was taken.
Journal and resume
A deployment writes the journal of its run next to the state, as
<environment>.journal.json, each time an item ends: what it did to the
item, and the hash and folder it sent. When a run fails, or dies, before
it records the state, the next run knows what it already sent, and skips
each item still in the workspace that has not changed since:
NOOP Orders.Notebook SOURCE_CHANGED: Definition and folder unchanged since an interrupted run sent it at 2026-09-26T10:04:12Z.
UPDATE Daily.DataPipeline SOURCE_CHANGED
- The state is still recorded only when a whole run succeeds; the journal never takes its place. The run that succeeds records what the interrupted runs sent, too.
- The journal holds the last run. A run that resumes another carries its entries over, so they are not lost if it is interrupted too.
- Each item costs one small write of the journal, a request of its own in OneLake. A journal that cannot be written costs only the resume: the run warns and goes on.
plan_all_items()shows the resume; it writes no journal.
Other backends
Any object with load(environment) and save(environment, state) is a
DeploymentStateBackend. One that also has lock(environment) and
force_unlock(environment) is a LockingStateBackend, and deployments
hold its lock; one with load_journal(environment) and
save_journal(environment, journal) is a JournalingStateBackend, and
deployments keep their journal there. One without them works as well,
without a lock or a resume.
Reference
Deployment state: what was last deployed successfully to an environment.
A DeploymentState records the workspace an environment targets, the
last commit deployed successfully for each item type, so the next
deployment of a type compares only what changed since then, and the hash
and folder last sent for each item, so an item sent unchanged needs
nothing. It is written only after a deployment in which every item
succeeded, and it never holds secrets.
DeploymentStateBackend is where states are kept; the engine knows
nothing more. LocalJsonStateBackend keeps them as JSON files in a local
folder, and OneLakeStateBackend, in pyfabricops.helpers.onelake_state,
in the Files of a lakehouse, where they outlive any CI run.
A backend that also locks (LockingStateBackend) keeps two runs from
deploying to one environment at a time: a run holds a DeploymentLock
from before it reads the state until after it records it. A lock has a
validity, after which another run may take it over, so a run that died
holding it does not block the environment for good.
A backend that also keeps journals (JournalingStateBackend) holds the
DeploymentJournal of the last run of each environment: what it did,
item by item, written as it went. The state is still recorded only when a
whole run succeeds; the journal tells the next run what a run that failed,
or died, already sent.
DeploymentJournal
dataclass
The journal of a deployment run: what it did, item by item.
A run writes its journal as it goes, next to the state, so that when it fails, or dies, before it records the state, the next run knows what it already sent. The journal never takes the place of the state, which is recorded only when a whole run succeeds. It holds no secret.
Attributes:
| Name | Type | Description |
|---|---|---|
environment |
str
|
The environment the run deployed to. |
workspace |
str
|
The workspace name or ID, as given to
|
run_id |
str
|
A random ID of the run. |
source_commit |
str
|
The commit the run deployed. |
started_at_utc |
str
|
When the run started, as
|
entries |
Sequence[JournalEntry]
|
What each action did, in the order the actions ended; the first ones may come from the interrupted run this one resumed. Stored as a tuple. |
finished_at_utc |
str | None
|
When the run ended, or None while it runs, or when it died. |
ok |
bool | None
|
Whether every item succeeded, once the run ended. |
Source code in src/pyfabricops/helpers/deployment_state.py
368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 | |
interrupted
property
Whether the run failed, or ended before finishing.
finish(ok)
Mark the run ended.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ok
|
bool
|
Whether every item succeeded. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
DeploymentJournal |
DeploymentJournal
|
A journal with its finish. |
Source code in src/pyfabricops/helpers/deployment_state.py
456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 | |
from_dict(data, *, source)
classmethod
Read a journal from JSON data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
Any
|
The parsed JSON. |
required |
source
|
str
|
Where the data came from, for error messages. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
DeploymentJournal |
DeploymentJournal
|
The journal. |
Raises:
| Type | Description |
|---|---|
ConfigurationError
|
If the data is not a journal. |
Source code in src/pyfabricops/helpers/deployment_state.py
508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 | |
sent()
Return what the journaled runs sent successfully, by item.
Returns:
| Type | Description |
|---|---|
dict[tuple[str, str], DeployedItem]
|
dict[tuple[str, str], DeployedItem]: For each item created, updated or moved, the hash and folder it was sent with, as of its last entry; an item deleted since is left out. |
Source code in src/pyfabricops/helpers/deployment_state.py
472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 | |
start(environment, workspace, source_commit, *, resumed=None)
classmethod
Begin the journal of a run.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
environment
|
str
|
The environment of the run. |
required |
workspace
|
str
|
The workspace, as given to |
required |
source_commit
|
str
|
The commit the run deploys. |
required |
resumed
|
DeploymentJournal
|
The journal of the interrupted run this one resumes: its entries are carried over, so that what it sent is not lost if this run is interrupted too. |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
DeploymentJournal |
DeploymentJournal
|
The journal, with no finish yet. |
Source code in src/pyfabricops/helpers/deployment_state.py
406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 | |
to_dict()
Return the journal as JSON-ready data.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
dict[str, Any]: The fields, with each entry as a dict. |
Source code in src/pyfabricops/helpers/deployment_state.py
494 495 496 497 498 499 500 501 502 503 504 505 506 | |
with_entry(entry)
Add what one more action did.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
entry
|
JournalEntry
|
The entry. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
DeploymentJournal |
DeploymentJournal
|
A journal with the entry last. |
Source code in src/pyfabricops/helpers/deployment_state.py
444 445 446 447 448 449 450 451 452 453 454 | |
DeploymentLock
dataclass
The lock a deployment run holds on the state of an environment.
A lock holds no secret: who took it, when, and until when it holds.
Attributes:
| Name | Type | Description |
|---|---|---|
environment |
str
|
The environment whose state is locked. |
lock_id |
str
|
A random ID, which tells this lock from any other. |
holder |
str
|
Who took it: |
acquired_at_utc |
str
|
When it was taken, as
|
expires_at_utc |
str
|
When another run may take it over, if its run has not released it. |
Source code in src/pyfabricops/helpers/deployment_state.py
232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 | |
expired
property
Whether another run may take the lock over.
describe()
Say who holds the lock, since when and until when.
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
Such as |
Source code in src/pyfabricops/helpers/deployment_state.py
282 283 284 285 286 287 288 289 290 291 292 293 | |
from_dict(data, *, source)
classmethod
Read a lock from JSON data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
Any
|
The parsed JSON. |
required |
source
|
str
|
Where the data came from, for error messages. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
DeploymentLock |
DeploymentLock
|
The lock. |
Raises:
| Type | Description |
|---|---|
ConfigurationError
|
If the data is not a lock. |
Source code in src/pyfabricops/helpers/deployment_state.py
304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 | |
new(environment, ttl_seconds)
classmethod
Make a lock for this process, valid for ttl_seconds.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
environment
|
str
|
The environment to lock. |
required |
ttl_seconds
|
float
|
How long the lock holds unless released. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
DeploymentLock |
DeploymentLock
|
The lock, not taken yet. |
Source code in src/pyfabricops/helpers/deployment_state.py
256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 | |
to_dict()
Return the lock as JSON-ready data.
Returns:
| Type | Description |
|---|---|
dict[str, str]
|
dict[str, str]: The fields. |
Source code in src/pyfabricops/helpers/deployment_state.py
295 296 297 298 299 300 301 302 | |
DeploymentState
dataclass
The last successful deployment to an environment.
Attributes:
| Name | Type | Description |
|---|---|---|
environment |
str
|
The environment the state is kept under. |
workspace |
str
|
The workspace name or ID the deployments target, as
given to |
workspace_id |
str
|
The resolved workspace ID. |
source_commit |
str
|
The commit the last successful deployment deployed. |
commits |
Mapping[str, str]
|
The last commit deployed successfully for each item type: where its next deployment compares from. Read-only. |
deployed_at_utc |
str
|
When the last successful deployment finished,
as |
items |
Mapping[tuple[str, str], DeployedItem]
|
What was last sent
for each item, by |
schema_version |
int
|
The layout of the stored state. |
Source code in src/pyfabricops/helpers/deployment_state.py
80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 | |
from_dict(data, *, source)
classmethod
Read a state from JSON data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
Any
|
The parsed JSON. |
required |
source
|
str
|
Where the data came from, for error messages. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
DeploymentState |
DeploymentState
|
The state. |
Raises:
| Type | Description |
|---|---|
ConfigurationError
|
If the data is not a state this version of pyfabricops can read. |
Source code in src/pyfabricops/helpers/deployment_state.py
153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 | |
targets(workspace)
Tell whether the state was recorded for a workspace.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
workspace
|
str
|
A workspace name or ID. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
bool |
bool
|
True when it is the workspace given or its resolved ID. |
Source code in src/pyfabricops/helpers/deployment_state.py
118 119 120 121 122 123 124 125 126 127 128 | |
to_dict()
Return the state as JSON-ready data.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
dict[str, Any]: The fields, with |
Source code in src/pyfabricops/helpers/deployment_state.py
130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 | |
DeploymentStateBackend
Bases: Protocol
Where deployment states are kept, one per environment.
Any object with these two methods is a backend, so states can live in a local folder, a blob container or a pipeline artifact without the engine knowing. A backend must never store secrets; a state has none.
Source code in src/pyfabricops/helpers/deployment_state.py
578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 | |
load(environment)
Return the state of an environment, or None when it has none.
Source code in src/pyfabricops/helpers/deployment_state.py
587 588 589 | |
save(environment, state)
Store the state of an environment, replacing the previous one.
Source code in src/pyfabricops/helpers/deployment_state.py
591 592 593 | |
JournalEntry
dataclass
What one action of a deployment run did.
Attributes:
| Name | Type | Description |
|---|---|---|
item_type |
str
|
The Fabric item type. |
display_name |
str
|
The item display name. |
outcome |
str
|
|
at_utc |
str
|
When the action ended, as |
content_hash |
str | None
|
The hash of the definition the action sent, when known. |
folder_path |
str | None
|
The workspace folder the item belongs in, or None for the workspace root. |
Source code in src/pyfabricops/helpers/deployment_state.py
343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 | |
JournalingStateBackend
Bases: DeploymentStateBackend, Protocol
A state backend that also keeps the journal of each environment's last run.
deploy_all_items writes the journal as each item ends. The backends
of pyfabricops keep journals; a backend without these methods still
works, and a run that fails is then not resumed.
Source code in src/pyfabricops/helpers/deployment_state.py
621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 | |
load_journal(environment)
Return the journal of the last run, or None without one.
Source code in src/pyfabricops/helpers/deployment_state.py
632 633 634 | |
save_journal(environment, journal)
Store the journal of a run, replacing the one before.
Source code in src/pyfabricops/helpers/deployment_state.py
636 637 638 639 640 | |
LocalJsonStateBackend
Keep deployment states as JSON files in a local folder.
Each environment gets <folder>/<environment>.json. In the file name,
characters other than letters, digits, ., _ and - become
-; the environment is also stored inside the file, so two names that
map to one file are caught. A file is replaced atomically, so a failed
write leaves the previous state intact.
The lock of an environment is <folder>/<environment>.lock, created
only when there is none. It says who holds it and until when; once
expired, another run takes it over. It keeps apart runs that share the
folder, such as two on one machine. The journal of the environment's
last run is <folder>/<environment>.journal.json.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
folder
|
str | Path
|
The folder of the state files, created on the first save. |
required |
lock_timeout
|
float
|
How many seconds to wait for a lock another run holds. Defaults to 0: fail at once. |
0
|
lock_ttl
|
float
|
How many seconds a lock holds unless its run releases it. Defaults to two hours. |
LOCK_TTL_SECONDS
|
Examples:
state = LocalJsonStateBackend('.pyfabricops/state')
report = deploy_all_items(
'Sales-PRD',
'workspace',
start_path='workspace',
state_backend=state,
environment='prod',
)
Source code in src/pyfabricops/helpers/deployment_state.py
643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 | |
force_unlock(environment)
Remove the lock of an environment, whoever holds it.
For a lock whose run is gone, when waiting for it to expire is not an option. Make sure no run is deploying to the environment first.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
environment
|
str
|
The environment name. |
required |
Returns:
| Type | Description |
|---|---|
DeploymentLock | None
|
DeploymentLock | None: The lock removed, or None without one. |
Source code in src/pyfabricops/helpers/deployment_state.py
769 770 771 772 773 774 775 776 777 778 779 780 781 782 | |
load(environment)
Return the state of an environment, or None when it has none.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
environment
|
str
|
The environment name. |
required |
Returns:
| Type | Description |
|---|---|
DeploymentState | None
|
DeploymentState | None: The state, or None without a file. |
Raises:
| Type | Description |
|---|---|
ConfigurationError
|
If the file is not a valid state, or holds the state of another environment. |
Source code in src/pyfabricops/helpers/deployment_state.py
692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 | |
load_journal(environment)
Return the journal of the environment's last run, if any.
A file that holds no journal of the environment is left alone with a warning: a journal only saves work, and its loss costs none.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
environment
|
str
|
The environment name. |
required |
Returns:
| Type | Description |
|---|---|
DeploymentJournal | None
|
DeploymentJournal | None: The journal, or None. |
Source code in src/pyfabricops/helpers/deployment_state.py
784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 | |
lock(environment)
Hold the lock of an environment for the length of a with block.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
environment
|
str
|
The environment name. |
required |
Returns:
| Type | Description |
|---|---|
AbstractContextManager[DeploymentLock]
|
AbstractContextManager[DeploymentLock]: Gives the lock held, and releases it when the block ends, even on an error. |
Raises:
| Type | Description |
|---|---|
DeploymentLockedError
|
On entering the block, if another run
holds the lock and still does after |
Source code in src/pyfabricops/helpers/deployment_state.py
746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 | |
save(environment, state)
Store the state of an environment, replacing the previous one.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
environment
|
str
|
The environment name. |
required |
state
|
DeploymentState
|
The state to store. |
required |
Raises:
| Type | Description |
|---|---|
ConfigurationError
|
If the state belongs to another environment. |
OSError
|
If the file cannot be written. |
Source code in src/pyfabricops/helpers/deployment_state.py
724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 | |
save_journal(environment, journal)
Store the journal of a run, replacing the one before.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
environment
|
str
|
The environment name. |
required |
journal
|
DeploymentJournal
|
The journal. |
required |
Raises:
| Type | Description |
|---|---|
OSError
|
If the file cannot be written. |
Source code in src/pyfabricops/helpers/deployment_state.py
804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 | |
LockingStateBackend
Bases: DeploymentStateBackend, Protocol
A state backend that also locks the state of an environment.
deploy_all_items holds the lock from before it reads the state until
after it records it, so two runs never deploy to one environment at a
time. The backends of pyfabricops lock; a backend with only load
and save still works, without a lock.
Source code in src/pyfabricops/helpers/deployment_state.py
596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 | |
force_unlock(environment)
Remove the lock of an environment, whoever holds it.
Source code in src/pyfabricops/helpers/deployment_state.py
616 617 618 | |
lock(environment)
Hold the lock of an environment for the length of a with block.
Raises:
| Type | Description |
|---|---|
DeploymentLockedError
|
If another run holds it. |
Source code in src/pyfabricops/helpers/deployment_state.py
607 608 609 610 611 612 613 614 | |
journal_json(journal)
Write a journal as the JSON its file holds.
Source code in src/pyfabricops/helpers/deployment_state.py
1020 1021 1022 | |
state_file_name(environment)
Name the files of an environment's state and lock, without extension.
Characters other than letters, digits, ., _ and - become
-.
Raises:
| Type | Description |
|---|---|
ConfigurationError
|
If nothing of the name is left. |
Source code in src/pyfabricops/helpers/deployment_state.py
1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 | |
Deployment states kept in a OneLake lakehouse.
OneLakeStateBackend keeps the state of each environment as a JSON file in
the Files of a lakehouse, and its lock next to it, through the OneLake Blob
API. Every write is conditional: a state is saved only over the one the run
read, and a lock is created only where there is none, so two runs never
overwrite each other. It needs a token for OneLake, which accepts only the
Azure Storage audience, and no dependency beyond requests.
OneLakeStateBackend
Keep deployment states as JSON files in a OneLake lakehouse.
Each environment gets Files/<folder>/<environment>.json in the
lakehouse, named as LocalJsonStateBackend names its files, and its
lock <environment>.lock and the journal of its last run
<environment>.journal.json next to it. The lakehouse can be in any
workspace the identity can write to, such as one kept for operations.
A state is saved only over the one load read, or where there was
none: when another run saved one in between, the save fails instead of
overwriting it. A lock is created only where there is none; it says who
holds it and until when, and once expired another run takes it over.
The workspace and the lakehouse are looked up once, by name or ID, and
then reached by ID. The identity needs a token for OneLake, which
set_auth_provider gets as for the Fabric API.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
workspace
|
str
|
The name or ID of the lakehouse's workspace. |
required |
lakehouse
|
str
|
The name or ID of the lakehouse. |
required |
folder
|
str
|
The folder under |
'pyfabricops/state'
|
endpoint
|
str
|
The OneLake Blob endpoint. Defaults to the
global one, |
ONELAKE_BLOB_ENDPOINT
|
lock_timeout
|
float
|
How many seconds to wait for a lock another run holds. Defaults to 0: fail at once. |
0
|
lock_ttl
|
float
|
How many seconds a lock holds unless its run releases it. Defaults to two hours. |
LOCK_TTL_SECONDS
|
Examples:
state = OneLakeStateBackend('Ops', 'DeploymentState')
report = deploy_all_items(
'Sales-PRD',
staging,
start_path=staging,
repository_path='workspace',
state_backend=state,
environment='prod',
)
Source code in src/pyfabricops/helpers/onelake_state.py
56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 | |
force_unlock(environment)
Remove the lock of an environment, whoever holds it.
For a lock whose run is gone, when waiting for it to expire is not an option. Make sure no run is deploying to the environment first.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
environment
|
str
|
The environment name. |
required |
Returns:
| Type | Description |
|---|---|
DeploymentLock | None
|
DeploymentLock | None: The lock removed, or None without one. |
Source code in src/pyfabricops/helpers/onelake_state.py
284 285 286 287 288 289 290 291 292 293 294 295 296 297 | |
load(environment)
Return the state of an environment, or None when it has none.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
environment
|
str
|
The environment name. |
required |
Returns:
| Type | Description |
|---|---|
DeploymentState | None
|
DeploymentState | None: The state, or None without a file. |
Raises:
| Type | Description |
|---|---|
ConfigurationError
|
If the file is not a valid state, or holds the state of another environment, or the lakehouse is not found. |
RequestError
|
If OneLake cannot be read. |
Source code in src/pyfabricops/helpers/onelake_state.py
124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 | |
load_journal(environment)
Return the journal of the environment's last run, if any.
A file that holds no journal of the environment is left alone with a warning: a journal only saves work, and its loss costs none.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
environment
|
str
|
The environment name. |
required |
Returns:
| Type | Description |
|---|---|
DeploymentJournal | None
|
DeploymentJournal | None: The journal, or None. |
Raises:
| Type | Description |
|---|---|
RequestError
|
If OneLake cannot be read. |
Source code in src/pyfabricops/helpers/onelake_state.py
213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 | |
lock(environment)
Hold the lock of an environment for the length of a with block.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
environment
|
str
|
The environment name. |
required |
Returns:
| Type | Description |
|---|---|
AbstractContextManager[DeploymentLock]
|
AbstractContextManager[DeploymentLock]: Gives the lock held, and releases it when the block ends, even on an error. |
Raises:
| Type | Description |
|---|---|
DeploymentLockedError
|
On entering the block, if another run
holds the lock and still does after |
Source code in src/pyfabricops/helpers/onelake_state.py
261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 | |
save(environment, state)
Store the state of an environment, over the one load read.
Without an earlier load of the environment, the state is written
whatever the file holds.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
environment
|
str
|
The environment name. |
required |
state
|
DeploymentState
|
The state to store. |
required |
Raises:
| Type | Description |
|---|---|
ConfigurationError
|
If the state belongs to another environment, or the lakehouse is not found. |
RequestError
|
If another run saved a state since |
Source code in src/pyfabricops/helpers/onelake_state.py
160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 | |
save_journal(environment, journal)
Store the journal of a run, replacing the one before.
Only the run that holds the lock writes the journal, so it is written whatever the file holds.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
environment
|
str
|
The environment name. |
required |
journal
|
DeploymentJournal
|
The journal. |
required |
Raises:
| Type | Description |
|---|---|
RequestError
|
If OneLake cannot be written. |
Source code in src/pyfabricops/helpers/onelake_state.py
236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 | |