You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 12:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Thu Sep 18, 2025 00:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
286085921020,5sleep00-21swapper/000:25:200
28340722930,3chrt2834067-21cpu23:10:162
28340722930,3chrt2834067-21cpu23:10:152
274962028268,9sleep20-21swapper/219:09:162
274962028268,9sleep20-21swapper/219:09:152
274950526848,14sleep30-21swapper/319:07:483
274950526848,14sleep30-21swapper/319:07:483
162399670,58rtkit-daemon0-21swapper/019:08:110
162399670,58rtkit-daemon0-21swapper/019:08:110
162399650,45rtkit-daemon1753-21gnome-shell23:02:012
2749841996311,47cyclictest1622-21rtkit-daemon21:20:210
274984599613,11cyclictest1938-21gsd-housekeepin23:46:041
274942626141,14sleep10-21swapper/119:06:461
274942626141,14sleep10-21swapper/119:06:461
27786872590,5sleep3411rcuc/320:25:313
2749851995820,11cyclictest2793330-21basename21:10:173
2749841995824,7cyclictest13-21ksoftirqd/022:03:220
274984599565,28cyclictest2848862-21sed23:50:241
274984599565,23cyclictest2790229-21/usr/sbin/munin21:00:121
274984599545,27cyclictest2785373-40lsb_release20:45:211
2749851995322,26cyclictest1753-21gnome-shell23:23:413
274984599538,40cyclictest1622-21rtkit-daemon23:33:421
274984599535,28cyclictest1753-21gnome-shell22:13:541
274984599535,28cyclictest1753-21gnome-shell22:13:531
274984199535,44cyclictest1622-21rtkit-daemon22:39:340
2749841995321,16cyclictest13-21ksoftirqd/020:26:350
274985199524,21cyclictest2798835-21awk21:25:253
2749851995219,12cyclictest2791709-21apt-get21:05:003
2749851995218,11cyclictest2750957-21sed19:10:313
274985199521,20cyclictest1753-21gnome-shell23:18:333
274984599526,22cyclictest2840181-21/usr/sbin/munin23:25:241
274984599522,16cyclictest1734-21gvfs-udisks2-vo00:31:561
2749845995211,19cyclictest2758160-21taskset19:30:201
2749841995224,16cyclictest13-21ksoftirqd/022:09:470
27603492510,6sleep32197014-21PK-Backend19:35:283
274985199512,44cyclictest1753-21gnome-shell00:05:143
2749851995122,25cyclictest1753-21gnome-shell19:31:263
2749851995121,10cyclictest42-21ksoftirqd/323:48:373
2749851995116,10cyclictest2805443-21expr21:45:133
274985199511,43cyclictest854-21avahi-daemon19:17:533
2749851995112,13cyclictest438-20systemd-journal20:59:553
2749851995110,37cyclictest0-21swapper/321:22:113
274985199509,18cyclictest1753-21gnome-shell22:20:343
274985199502,43cyclictest1753-21gnome-shell21:31:323
2749851995019,11cyclictest2826929-21grep22:50:143
2749851995010,17cyclictest1753-21gnome-shell22:58:293
2749851995010,16cyclictest1753-21gnome-shell20:20:003
274984599504,27cyclictest1753-21gnome-shell19:16:441
274984599504,17cyclictest1753-21gnome-shell23:21:081
274984599500,27cyclictest1753-21gnome-shell00:16:471
274984199501,45cyclictest1622-21rtkit-daemon20:57:590
27880462490,6sleep013-21ksoftirqd/020:52:530
274985199492,8cyclictest1753-21gnome-shell22:02:333
274985199492,43cyclictest1753-21gnome-shell19:51:093
2749851994922,8cyclictest1753-21gnome-shell19:20:323
2749851994922,8cyclictest1753-21gnome-shell19:20:313
274985199492,27cyclictest1753-21gnome-shell00:38:193
274985199492,26cyclictest1753-21gnome-shell21:38:323
2749851994912,12cyclictest2770992-21proc_pri20:05:283
274984599497,14cyclictest1753-21gnome-shell00:00:321
274984599495,18cyclictest1753-21gnome-shell23:19:341
274984599495,18cyclictest1753-21gnome-shell19:38:101
274984599494,20cyclictest2843398-21df23:35:161
274984599494,19cyclictest2797269-21cat21:20:291
274984599493,26cyclictest2765442-21ntp_states19:50:281
274984599493,26cyclictest2753709-21latency_hist19:20:021
274984599493,26cyclictest2753709-21latency_hist19:20:021
274984599493,23cyclictest2793611-21awk21:10:201
2749845994920,24cyclictest2778261-21cat20:25:211
274984199493,42cyclictest1622-21rtkit-daemon22:20:080
2749841994922,22cyclictest2821272-21cron22:35:010
2749851994837,7cyclictest2824783-21turbostat22:44:593
274985199482,42cyclictest1753-21gnome-shell19:45:323
2749851994822,22cyclictest2776032-21df_inode20:20:163
2749851994817,10cyclictest42-21ksoftirqd/323:07:373
2749851994816,9cyclictest2780405-21ls20:30:303
2749851994814,11cyclictest2808486-21cat21:55:273
2749851994813,11cyclictest2767004-21cat19:55:143
274985199481,22cyclictest2850939-21cat23:55:333
274984599487,20cyclictest2825042-21apt22:45:121
274984599485,20cyclictest2818736-21/usr/sbin/munin22:25:291
2749845994812,17cyclictest1753-21gnome-shell23:14:431
2749845994812,17cyclictest1753-21gnome-shell23:14:421
274984199488,8cyclictest13-21ksoftirqd/019:55:270
274984199482,29cyclictest13-21ksoftirqd/022:10:160
274984199482,29cyclictest13-21ksoftirqd/022:10:150
274985199479,18cyclictest1753-21gnome-shell20:02:373
274985199477,19cyclictest1753-21gnome-shell20:50:483
274985199477,17cyclictest1753-21gnome-shell21:08:323
274985199477,17cyclictest1753-21gnome-shell21:08:323
274985199477,16cyclictest1753-21gnome-shell22:27:273
274985199472,8cyclictest1753-21gnome-shell20:12:023
274985199472,20cyclictest2785361-40lsb_release20:45:203
2749851994717,7cyclictest2849691-21idleruntime-cro23:55:003
2749851994716,12cyclictest42-21ksoftirqd/323:12:573
2749851994716,12cyclictest42-21ksoftirqd/323:12:573
2749851994710,17cyclictest1753-21gnome-shell21:40:573
2749851994710,16cyclictest1753-21gnome-shell22:39:003
274984599477,19cyclictest1753-21gnome-shell21:18:351
274984599475,17cyclictest1753-21gnome-shell19:27:231
2749841994735,9cyclictest13-21ksoftirqd/020:05:490
2749841994726,9cyclictest13-21ksoftirqd/021:15:220
2749841994718,12cyclictest13-21ksoftirqd/021:30:060
162399470,33rtkit-daemon1753-21gnome-shell20:22:272
28434172460,6chrt0-21swapper/323:35:203
274985199468,18cyclictest1753-21gnome-shell00:11:333
274985199463,19cyclictest2811658-21ntpq22:05:263
274985199462,8cyclictest1753-21gnome-shell23:00:263
274985199462,17cyclictest2757269-31apt-check19:27:303
2749851994618,8cyclictest1753-21gnome-shell22:31:333
2749851994615,7cyclictest2814892-21kernelversion22:15:213
274985199461,25cyclictest1753-21gnome-shell21:17:513
274985199461,25cyclictest1753-21gnome-shell20:40:243
274985199461,22cyclictest1753-21gnome-shell00:25:123
2749851994610,16cyclictest1753-21gnome-shell19:43:133
274984599463,20cyclictest0-21swapper/120:09:151
2749845994625,17cyclictest2783315-21df20:40:151
2749841994617,12cyclictest13-21ksoftirqd/021:41:290
162399460,33rtkit-daemon1753-21gnome-shell00:10:132
28259192450,8sleep342-21ksoftirqd/322:45:293
274985199459,16cyclictest1753-21gnome-shell00:01:543
274985199458,18cyclictest1753-21gnome-shell23:31:013
2749851994519,7cyclictest854-21avahi-daemon00:22:583
2749851994517,8cyclictest1753-21gnome-shell00:30:333
2749851994510,16cyclictest1753-21gnome-shell20:39:073
2749847994515,13cyclictest141rcu_preempt23:44:352
274984599456,16cyclictest1753-21gnome-shell20:38:431
274984599451,26cyclictest854-21avahi-daemon21:54:331
2749845994510,30cyclictest2631682-21snapd22:32:481
274984199455,9cyclictest2841969-21strings23:30:210
2749841994534,8cyclictest13-21ksoftirqd/022:52:020
2749841994525,8cyclictest13-21ksoftirqd/000:10:370
2749841994524,8cyclictest13-21ksoftirqd/020:45:580
2749841994524,8cyclictest13-21ksoftirqd/020:23:330
2749841994523,8cyclictest13-21ksoftirqd/020:16:120
2749841994518,16cyclictest13-21ksoftirqd/022:56:110
274985199449,16cyclictest1753-21gnome-shell00:15:413
2749851994413,8cyclictest0-21swapper/322:10:383
2749851994413,8cyclictest0-21swapper/322:10:383
2749845994429,11cyclictest2827998-21systemd-udevd22:51:591
2749845994411,15cyclictest2856004-21/usr/sbin/munin00:10:301
28465872430,8chrt1753-21gnome-shell23:45:150
274985199437,16cyclictest1753-21gnome-shell23:29:103
274984599436,15cyclictest0-21swapper/121:59:021
274984599434,16cyclictest2850726-21taskset23:55:291
274984599434,16cyclictest2798084-21taskset21:25:121
2749845994318,10cyclictest2631710-21snapd22:02:481
274984199438,14cyclictest2197014-21PK-Backend23:43:390
274984199437,20cyclictest13-21ksoftirqd/019:25:070
2749841994324,7cyclictest13-21ksoftirqd/023:15:570
2749841994322,7cyclictest13-21ksoftirqd/021:37:590
28323982420,7sleep02197014-21PK-Backend23:05:190
28039422420,13sleep1251rcuc/121:40:231
274984599426,11cyclictest1753-21gnome-shell20:51:551
2749845994232,7cyclictest1-21systemd20:20:011
274984599423,16cyclictest1753-21gnome-shell23:43:301
274984599423,15cyclictest26-21ksoftirqd/123:09:221
2749841994232,7cyclictest161rcuc/021:14:120
2749841994232,7cyclictest13-21ksoftirqd/021:48:340
2749841994221,8cyclictest13-21ksoftirqd/023:38:580
2749841994221,8cyclictest13-21ksoftirqd/022:45:420
2749841994217,7cyclictest13-21ksoftirqd/000:24:570
2749841994212,14cyclictest13-21ksoftirqd/022:26:000
162399420,31rtkit-daemon860997-21snmpd00:18:432
162399420,30rtkit-daemon2197014-21PK-Backend00:25:242
2749847994124,12cyclictest438-20systemd-journal20:57:152
274984799411,9cyclictest2197014-21PK-Backend20:02:532
2749847994118,5cyclictest141rcu_preempt21:40:202
2749847994114,6cyclictest141rcu_preempt19:55:052
2749847994113,7cyclictest141rcu_preempt20:05:302
274984599414,16cyclictest26-21ksoftirqd/119:58:321
2749845994127,11cyclictest2789970-21awk21:00:001
274984599412,15cyclictest1753-21gnome-shell22:05:561
274984199417,13cyclictest13-21ksoftirqd/019:17:070
274984199416,10cyclictest2197014-21PK-Backend23:12:210
274984199416,10cyclictest2197014-21PK-Backend23:12:210
2749841994123,7cyclictest13-21ksoftirqd/000:31:380
2749841994122,7cyclictest13-21ksoftirqd/020:37:400
2749841994121,8cyclictest13-21ksoftirqd/023:29:080
2749841994121,8cyclictest13-21ksoftirqd/020:34:040
274984199412,12cyclictest13-21ksoftirqd/021:53:500
2749841994111,16cyclictest13-21ksoftirqd/020:01:520
162399410,29rtkit-daemon2197014-21PK-Backend21:54:383
162399410,29rtkit-daemon2197014-21PK-Backend21:01:412
162399410,28rtkit-daemon2197014-21PK-Backend00:24:142
274984799404,13cyclictest17531gnome-shell21:55:332
2749847994027,9cyclictest2794399-21pool-org.gnome.21:15:002
2749847994027,9cyclictest2784405-21apt-get20:45:102
2749847994025,11cyclictest2784400-21cron20:45:002
2749847994022,13cyclictest2841656-21df23:30:122
2749847994022,13cyclictest2823876-21memory22:40:242
2749847994021,8cyclictest1753-21gnome-shell19:43:352
274984799401,9cyclictest2197014-21PK-Backend19:53:422
274984799401,8cyclictest0-21swapper/200:06:362
2749847994017,19cyclictest2751472-21cron19:15:002
274984599409,26cyclictest2771901-21/usr/sbin/munin20:10:301
274984599407,9cyclictest0-21swapper/122:39:151
274984599404,16cyclictest1753-21gnome-shell21:36:421
274984599403,15cyclictest2768462-21cpu20:00:101
2749845994024,12cyclictest2831436-21sshd23:01:141
2749845994021,14cyclictest438-20systemd-journal22:20:001
2749845994011,24cyclictest2864261-21df00:35:151
2749845994010,26cyclictest2777037-21chrt20:23:321
274984199408,11cyclictest13-21ksoftirqd/021:26:050
274984199406,15cyclictest13-21ksoftirqd/022:16:490
2749841994031,6cyclictest13-21ksoftirqd/020:44:480
2749841994029,8cyclictest13-21ksoftirqd/021:07:100
2749841994029,8cyclictest13-21ksoftirqd/021:07:100
2749841994027,10cyclictest13-21ksoftirqd/000:05:090
2749841994022,14cyclictest2760074-21memory19:35:220
2749841994021,8cyclictest13-21ksoftirqd/019:45:100
2749841994020,8cyclictest13-21ksoftirqd/000:00:080
274984199401,16cyclictest13-21ksoftirqd/019:44:210
274984199401,13cyclictest2758708-21/usr/sbin/munin19:30:310
2749841994010,16cyclictest13-21ksoftirqd/000:39:590
2749851993918,4cyclictest0-21swapper/323:44:513
274984799392,8cyclictest1753-21gnome-shell23:45:402
2749847993915,9cyclictest2962346-21in:imuxsock22:49:592
274984599398,8cyclictest2830018-21perf23:00:011
274984599391,34cyclictest1903-21packagekitd19:48:311
2749845993910,25cyclictest2800143-21expr21:30:171
274984199398,15cyclictest13-21ksoftirqd/019:12:250
274984199396,18cyclictest13-21ksoftirqd/023:53:300
2749841993927,9cyclictest13-21ksoftirqd/019:50:280
274984199392,11cyclictest13-21ksoftirqd/021:57:170
2749841993919,8cyclictest13-21ksoftirqd/000:16:300
274984799383,19cyclictest1753-21gnome-shell19:45:582
274984799382,8cyclictest1753-21gnome-shell22:10:172
274984799382,8cyclictest1753-21gnome-shell22:10:172
274984799382,7cyclictest1753-21gnome-shell21:25:332
274984799382,7cyclictest1753-21gnome-shell20:26:092
2749847993827,7cyclictest28750irq/122-ahci[0023:21:562
2749847993826,8cyclictest1753-21gnome-shell23:27:512
2749847993823,11cyclictest2772050-21grep20:10:142
2749847993813,6cyclictest141rcu_preempt00:30:272
274984599385,9cyclictest26-21ksoftirqd/121:06:061
274984599385,9cyclictest26-21ksoftirqd/121:06:061
2749845993821,13cyclictest2859470-21awk00:20:281
274984199389,7cyclictest2197014-21PK-Backend21:00:040
274984199389,16cyclictest13-21ksoftirqd/023:00:090
274984199386,12cyclictest13-21ksoftirqd/022:44:180
162399380,27rtkit-daemon2197014-21PK-Backend21:47:592
274984799373,11cyclictest141rcu_preempt22:20:162
2749847993727,7cyclictest1753-21gnome-shell22:35:342
2749847993720,12cyclictest438-20systemd-journal21:05:122
2749847993720,12cyclictest438-20systemd-journal21:05:112
2749847993718,14cyclictest2836059-40apt-check23:15:132
2749847993718,14cyclictest2179-21gvfsd-metadata20:20:022
2749847993711,6cyclictest141rcu_preempt20:50:292
274984599377,14cyclictest0-21swapper/100:05:241
2749845993726,8cyclictest1069-21runrttasks19:13:391
2749845993720,12cyclictest2823293-21sshd22:40:141
2749845993711,22cyclictest2861113-21uname00:25:231
2749841993726,8cyclictest13-21ksoftirqd/023:20:360
2749841993726,8cyclictest13-21ksoftirqd/020:11:020
2749841993726,8cyclictest13-21ksoftirqd/019:23:240
2749841993726,8cyclictest13-21ksoftirqd/019:23:240
162399370,26rtkit-daemon2197014-21PK-Backend21:51:322
162399370,26rtkit-daemon2197014-21PK-Backend19:18:152
162399370,26rtkit-daemon0-21swapper/222:01:082
162399370,26rtkit-daemon0-21swapper/220:34:372
162399370,26rtkit-daemon0-21swapper/219:31:032
2749847993625,7cyclictest1753-21gnome-shell23:56:372
2749847993625,7cyclictest1753-21gnome-shell00:03:482
2749847993623,9cyclictest34-21ksoftirqd/221:35:112
274984799362,14cyclictest3242253-21nautilus22:33:072
2749847993620,11cyclictest2827417-21cat22:50:232
2749847993618,13cyclictest855-21dbus-daemon23:10:012
274984799361,5cyclictest141rcu_preempt19:39:322
2749847993611,7cyclictest141rcu_preempt21:15:312
274984799361,12cyclictest2197014-21PK-Backend23:39:322
274984599365,10cyclictest2817434-21runrttasks22:23:111
2749845993620,12cyclictest1753-21gnome-shell20:34:531
274984599361,10cyclictest0-21swapper/119:41:201
274984199367,16cyclictest13-21ksoftirqd/023:57:130
274984799356,12cyclictest1753-21gnome-shell19:20:132
274984799356,12cyclictest1753-21gnome-shell19:20:132
2749847993518,13cyclictest2829553-21cat22:55:332
2749847993518,13cyclictest1-21systemd22:10:012
2749847993518,12cyclictest2818616-21ntpq22:25:262
2749847993512,6cyclictest141rcu_preempt20:35:202
274984799351,10cyclictest2427520-21snapd00:37:182
2749847993510,7cyclictest141rcu_preempt21:20:152
274984599351,29cyclictest2805453-21df21:45:131
2749847993416,14cyclictest2800109-21cstates21:30:162
274984799334,10cyclictest34-21ksoftirqd/222:16:532
2749847993323,7cyclictest27572352sleep219:27:232
274984799329,5cyclictest141rcu_preempt23:50:562
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional